Controlling PDM: Reproducibility vs. Convenience.

[mforbes]

Recently I started trying to use pdm for managing project dependencies and have been stumbling because of what I interpret as two distinct and somewhat conflicting goals:

1. Default behaviour that causes pdm to try to guess what to do in a manner most convenient for users.
2. Reproducible behaviour where the user can predictable control what is going to happen.

I have been trying to achieve goal 2, but keep stumbling on issues related to 1. This discussing is to help document the issues I had, their resolution, with the goal of being turned into clear documentation so future users like me do not stumble on similar issues.

Use Case

I am trying to setup a general workflow for establishing a reproducible computing environment for projects with cross-platform support. I am currently using a Makefile to specify the workflow so that users can get up and running by doing something like this:

1. Clone the project to their computer.
2. Run make init.
3. Work with Jupyter Notebooks, develop, run tests, etc. in a reproducible environment.

My impression was that this should be possible with tools like Anaconda Project, Poetry and pdm through something like this:

Anaconda Project: (I am most familiar with this.)

anaconda-project prepare  # Not really needed... will be done as needed
anaconda-project run jupyter notebook
anaconda-project run pytest

Poetry adds a nice feature poetry shell which spawns a shell that one can work in. I can fake this with a custom command in Anaconda Project that runs bash using a local init script that activates the appropriate environment.

anaconda-project run shell

The reason I have to use a Makefile is that this all requires having Anaconda Project installed on the users computer which should not be taken for granted. I use this on CoCalc for example, which has everything installed, but where it needs to be activated by an appropriate command, and on HPC clusters where one needs to load an appropriate module etc. Even if conda is available, I don't want to muck up peoples environments, so I can't just tell them to do conda install anaconda-project. I can use the Makefile to code all of the subtleties so that the user just needs to do:

make init

and things work. (Plus, the Makefile then provides a reference about what needs to be done on various platforms.)

I have been trying to figure out how to do something similar with pdm but keep running into issues with semi-magic defaults (goal 1.) such as:

• The value of the interpreter specified in .pdm.toml. This is machine-dependent, so not something I should control in my project, but subtly overrides some of the explicit behaviour I was expecting (see python.use_venv = True does not cause packages to be installed in venv #973 (comment)). I think the correct way to control this is the -I, --ignore-python option, or setting PDM_IGNORE_SAVED_PYTHON=1 before running pdm, then using PDM_VENV_IN_PROJECT=1 and PDM_USE_VENV=1 works to create the isolated virtual environment in .venv... unless
• ... one has a conda environment active (pdm uses active conda virtualenv despite PDM_VENV_BACKEND=virtualenv #1756), in which case, that magically takes precedent over the flags. Here a workaround is to (un)set CONDA_PREFIX= before running pdm, or to explicitly create and use the virtualenv.
• To explicitly create the virtualenv in the project, I think one does

  pdm venv create [python]

  which can then be used as follows:

  pdm use -i .venv

  Note: the -i, --ignore-remembered flag is important because if you have run pdm use .venv before in another project, then that result will be cached somewhere (see the cache_dir output of pdm config) and reused. This is another semi-magical default behaviour introduced in Feature: Remember the last selection in "pdm use" #852 in response to Support configuring Python interpreter paths for different Python versions in .pdm.toml #846 which helps with goal 1. but creates a stumbling block for goal 2.

Thus, I am currently working with instructions and a Makefile with something like the following.

1. If needed, install pdm with pipx install pdm.

2. Run make init with the following Makefile

   PYTHON_VER ?= 3.11
   PRE ?= VIRTUAL_ENV= CONDA_PREFIX= PDM_VENV_IN_PROJECT=1 PDM_USE_VENV=1 PDM_VENV_BACKEND=virtualenv PDM_IGNORE_SAVED_PYTHON=1
   PDM ?= pdm
   RUN ?= $(PDM) run
   KERNEL ?= my-project
   
   init: .venv/bin/python$(PYTHON_VER)
       $(PRE) $(PDM) update
       $(PRE) $(PDM) install -G :all
   ifdef KERNEL
       $(PRE) $(RUN) python3 -m ipykernel install --user --name "$(KERNEL)" --display-name "Python 3 ($(KERNEL))"
   endif
   
   .venv/bin/python$(PYTHON_VER):
       $(PRE) $(PDM) venv create --force $(PYTHON_VER)
       $(PRE) $(PDM) use -i .venv
   
   clean:
   ifdef KERNEL
       -$(PRE) $(RUN) jupyter kernelspec uninstall -f "$(KERNEL)"
   endif
       -$(RM) -r __pypackage__ .venv
       -find . -name "__pycache__" -exec $(RM) -r {} +
       -find . -name ".ipynb_checkpoints" -exec $(RM) -r {} +
       -$(RM) -r _htmlcov .coverage .pytest_cache
   
   .PHONY: clean init

   To show some of the things I am trying to do (and why I probably need a Makefile) I have included instructions to register and unregister a Jupyter kernel for the current environment.

Is this overkill? Have I missed any other potential issues (have not extensively tested this yet).

Also, is there any way to get a command like pdm shell to work like poetry shell: i.e. spawning a shell with the appropriate venv activated? I know there is pdm-shell, but this just sets the environmental variables eval "$(pdm shell)": I would like to spawn a new shell like poetry shell does. With Anaconda Project one can define custom commands that do this anaconda-project run shell. Is there something similar with pdm? (I am sure I could probably do this with a plugin... is that the best approach or am I missing something.)

Suggestions/comments appreciated!

[mforbes]

Another issue I am having is trying to use conda on a Mac ARM platform under rosetta. Typically what one would do is set CONDA_SUBDIR=osx-64 before running conda. This works, but in order to get things compiling, I have to re-install pdm in that environment, otherwise it compiles against the arm version. Here is an example:

# Minimal pyproject.toml
[project]
requires-python = ">=3.10"

$ CONDA_SUBDIR=osx-64 CONDA_PREFIX= PDM_VENV_IN_PROJECT=1 PDM_USE_VENV=1 PDM_VENV_BACKEND=conda PDM_IGNORE_SAVED_PYTHON=1 pdm venv create 3.10
Virtualenv .../.venv is created successfully
$ pdm add pyfftw
Adding packages to default dependencies: pyfftw
Virtualenv .../.venv is reused.
...
  ✔ Install numpy 1.24.2 successful
  ✖ Install pyfftw 0.13.1 failed
Retry failed jobs
  ✖ Install pyfftw 0.13.1 failed
  * The Python version is: Python3.10 from ".../.venv/bin/python"
  * The NumPy version is: "1.21.6"

Why is NumPy version 1.21.6 being used if the installed version is 1.24.2? The debug log seems to show that pdm is installing this (probably with the wrong architecture).

If I install pdm in the virtual environment and run it from there, everything works, but this seems completely counter to the philosophy that PDM should be installed as a tool outside of the project.

$ conda activate .venv
$ pip install pdm
...
$ pdm add pyfftw
...
  ✔ Install pyfftw 0.13.1 successful
  ✔ Install numpy 1.24.2 successful
Installing the project as an editable package...
  ✔ Install test 0.0.1 successful

🎉 All complete!

Full traceback and logs

$ CONDA_SUBDIR=osx-64 CONDA_PREFIX= PDM_VENV_IN_PROJECT=1 PDM_USE_VENV=1 PDM_VENV_BACKEND=conda PDM_IGNORE_SAVED_PYTHON=1 pdm venv create 3.10
Virtualenv .../.venv is created successfully
$ pdm add pyfftw
Adding packages to default dependencies: pyfftw
Virtualenv .../.venv is reused.
🔒 Lock successful
...
  ✔ Install numpy 1.24.2 successful
  ✖ Install pyfftw 0.13.1 failed
Retry failed jobs
  ✖ Install pyfftw 0.13.1 failed

ERRORS:
add pyfftw failed:
Traceback (most recent call last):
  File "/data/apps/conda_arm64/lib/python3.9/concurrent/futures/thread.py", line 58, in run
...
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/builders/base.py", line 107, in log_subprocessor
    raise build_error(e) from None
pdm.exceptions.BuildError: Build backend raised error: Showing the last 10 lines of the build output:
Please note and check the following:

  * The Python version is: Python3.10 from ".../.venv/bin/python"
  * The NumPy version is: "1.21.6"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/var/folders/m7/dnr91tjs4gn58_t3k8zp_g000000gr/T/pdm-build-env-yo2qn11z-shared/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): tried: 
'/usr/local/cuda/lib/_multiarray_umath.cpython-310-darwin.so' (no such file), 
'/var/folders/m7/dnr91tjs4gn58_t3k8zp_g000000gr/T/pdm-build-env-yo2qn11z-shared/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so' (mach-o file, but is an incompatible 
architecture (have (arm64), need (x86_64))), '/usr/local/cuda/lib/_multiarray_umath.cpython-310-darwin.so' (no such file), 
'/private/var/folders/m7/dnr91tjs4gn58_t3k8zp_g000000gr/T/pdm-build-env-yo2qn11z-shared/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so' (mach-o file, but is an incompatible 
architecture (have (arm64), need (x86_64)))


See /var/folders/m7/dnr91tjs4gn58_t3k8zp_g000000gr/T/pdm-install-iaborwk7.log for detailed debug log.
...

pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: Saving wheel to cache: <Link https://files.pythonhosted.org/packages/9f/bc/7376df0393e816b60bdc627fc8f653706f4caa95bcf79d07302a672f893e/pyFFTW-0.13.1.tar.gz (from None)>
pdm.termui: Running PEP 517 backend to build a wheel for <Link https://files.pythonhosted.org/packages/9f/bc/7376df0393e816b60bdc627fc8f653706f4caa95bcf79d07302a672f893e/pyFFTW-0.13.1.tar.gz (from None)>
pdm.termui: Skipping requirement numpy==1.19.5; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy': mismatching marker python_version == "3.8" and platform_machine == "aarch64" and platform_python_implementation != "PyPy"
pdm.termui: Skipping requirement numpy==1.21.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin': mismatching marker python_version == "3.8" and platform_machine == "arm64" and platform_system == "Darwin"
pdm.termui: Skipping requirement numpy==1.21.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin': mismatching marker python_version == "3.9" and platform_machine == "arm64" and platform_system == "Darwin"
pdm.termui: Skipping requirement numpy==1.22.0; platform_machine=='loongarch64': mismatching marker platform_machine == "loongarch64"
pdm.termui: Skipping requirement numpy==1.22.3; python_version=='3.10' and platform_system=='Windows' and platform_python_implementation != 'PyPy': mismatching marker python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"
pdm.termui: Skipping requirement numpy==1.19.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_machine!='loongarch64' and platform_python_implementation != 'PyPy': mismatching marker python_version == "3.8" and (platform_machine != "arm64" or platform_system != "Darwin") and platform_machine != "aarch64" and platform_machine != "loongarch64" and platform_python_implementation != "PyPy"
pdm.termui: Skipping requirement numpy==1.19.5; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='loongarch64' and platform_python_implementation != 'PyPy': mismatching marker python_version == "3.9" and (platform_machine != "arm64" or platform_system != "Darwin") and platform_machine != "loongarch64" and platform_python_implementation != "PyPy"
pdm.termui: Skipping requirement numpy==1.23.2; python_version=='3.11' and platform_python_implementation != 'PyPy': mismatching marker python_version == "3.11" and platform_python_implementation != "PyPy"
pdm.termui: Skipping requirement numpy; python_version>='3.12': mismatching marker python_version >= "3.12"
pdm.termui: Skipping requirement numpy; python_version>='3.9' and platform_python_implementation=='PyPy': mismatching marker python_version >= "3.9" and platform_python_implementation == "PyPy"
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   numpy==1.21.6; python_version == "3.10" and (platform_system != "Windows" and platform_machine != "loongarch64") and platform_python_implementation != "PyPy"
pdm.termui:   setuptools<=59.4.0
pdm.termui:   wheel
pdm.termui:   Cython>=0.29.18
pdm.termui:   python>=3.10.9,<3.10.10
pdm.termui:   Adding requirement numpy==1.21.6; python_version == "3.10" and (platform_system != "Windows" and platform_machine != "loongarch64") and platform_python_implementation != "PyPy"

Is this a bug? How do I control the platform and CONDA_SUBDIR specification with PDM?

[mforbes]

A related question: why can't I specify the cache_dir in a local config file to ensure isolation. (I don't want a local pdm use to update my global cache).

$ pdm config -l cache_dir .local/pdm
[ValueError]: Config item 'cache_dir' is not allowed to set in project config.

Update: There is reference in #435 (comment) to assumptions being made about a centralized cache... but I still don't understand why, or how to get pdm use to not pollute the global cache. Maybe pdm use -i should not use the cache at all?

[bluss]

Hi, sorry if this is too tangential, but I use https://github.com/bluss/pyproject-local-kernel for reproducible notebooks - each directory of notebooks have their dependencies defined by a pyproject.toml. I use it with Rye, but it supports Pdm as well.