PDM fails to resolve on packages which have old versions with legacy version identifiers #1554

lambda · 2022-12-08T01:00:22Z

I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

Install PDM into a virtualenv using pip install pdm
Use pdm update -v to update dependencies on a package that depends on google-api-python-client

Actual behavior

PDM errors out with an inability to parse this legacy version specifier.

$ pdm update -v $EXTRA_PDM_UPDATE_ARGS --no-sync
python.use_venv is on, creating a virtualenv for this project...
Run command: ['/usr/local/bin/python', '-m', 'virtualenv', '/builds/software/py-beta-data-warehouse/.venv', '-p', '/usr/local/bin/python', '--no-pip', '--no-setuptools', '--no-wheel', '--prompt=py-beta-data-warehouse-3.9']
created virtual environment CPython3.9.15.final.0-64 in 63ms
  creator CPython3Posix(dest=/builds/software/py-beta-data-warehouse/.venv, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Virtualenv is created successfully at /builds/software/py-beta-data-warehouse/.venv
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   apache-beam[gcp]>=2.22.0

... snip ...

pdm.termui:   Adding requirement google-api-python-client==2.13.0
Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 258, in main
    return Core().main(args)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 190, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 185, in main
    f(project, options)
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/commands/update.py", line 60, in handle
    actions.do_update(
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/actions.py", line 384, in do_update
    resolved = do_lock(
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/actions.py", line 95, in do_lock
    mapping, dependencies = resolve(
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 372, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/usr/local/lib/python3.9/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/providers.py", line 261, in matches_gen
    yield from super_find()
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/providers.py", line 150, in matches_gen
    candidates = self._find_candidates(reqs[0])
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/providers.py", line 132, in _find_candidates
    return self.repository.find_candidates(
  File "/usr/local/lib/python3.9/site-packages/pdm/models/repositories.py", line 150, in find_candidates
    cans = list(self._find_candidates(requirement))
  File "/usr/local/lib/python3.9/site-packages/pdm/models/repositories.py", line 338, in _find_candidates
    for c in finder.find_all_packages(requirement.project_name)
  File "/usr/local/lib/python3.9/site-packages/unearth/finder.py", line 241, in find_all_packages
    return sorted(
  File "/usr/local/lib/python3.9/site-packages/unearth/finder.py", line 192, in _sort_key
    parse_version(package.version) if package.version is not None else 0,
  File "/usr/local/lib/python3.9/site-packages/packaging/version.py", line 52, in parse
    return Version(version)
  File "/usr/local/lib/python3.9/site-packages/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '1.0beta5prerelease'

Expected behavior

PDM should be able to resolve the specified version, regardless of whether there is an older version that has an invalid version specifier.

Workaround

Pin the version of Packaging to 21.3, such as pip install pdm packaging==21.3.

Further information

According to pypa/packaging#530, legacy version identifiers like this have been deprecated for some time, and were expected to produce warnings. Checking older CI logs, I see no such warnings from PDM, so we simply ran into this problem with no warning.

The text was updated successfully, but these errors were encountered:

frostming · 2022-12-08T01:25:54Z

Yeah the new released packaging==22.0 breaks this. But going forward we are not likely to keep the version cap on that requirement. It should be fixed upstream, to normalize the version to 1.0b5.

frostming · 2022-12-08T01:28:19Z

Perhaps we can skip the packages with invalid version in unearth

pradyunsg · 2022-12-08T03:30:34Z

Checking older CI logs, I see no such warnings from PDM, so we simply ran into this problem with no warning.

Hmm... What if you run with PYTHONDEVMODE=1?

lambda · 2022-12-08T14:08:49Z

@pradyunsg If I enable PYTHONDEVMODE=1 in my CI pipeline in which I saw this issue, I see several warnings during my pip install pdm step, but not during the pdm update step.

The ones during the pip install pdm step also don't provide enough information to know what version specifier it had trouble with, or what to do about it.

The pip install pdm step:

$ pip install pdm~=$PDM_VERSION packaging==21.3
/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Looking in indexes: https://gitlab-ci-token:****@gitlab.beta.team/api/v4/projects/beta%2Fpypi/packages/pypi/simple
Collecting pdm~=2.3.1
  Downloading pdm-2.3.2-py3-none-any.whl (197 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 197.8/197.8 KB 4.4 MB/s eta 0:00:00
Collecting packaging==21.3
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 KB 1.7 MB/s eta 0:00:00
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 KB 5.2 MB/s eta 0:00:00
Collecting requests-toolbelt
  Downloading requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 KB 4.1 MB/s eta 0:00:00
/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/specifiers.py:255: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Collecting shellingham>=1.3.2
  Downloading shellingham-1.5.0-py2.py3-none-any.whl (9.3 kB)
  
  ... snip ...
  
  Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 KB 10.3 MB/s eta 0:00:00
Installing collected packages: resolvelib, msgpack, lockfile, distlib, commonmark, zipp, urllib3, tomlkit, tomli, shellingham, python-dotenv, pyparsing, pygments, platformdirs, installer, idna, filelock, charset-normalizer, certifi, blinker, virtualenv, rich, requests, pyproject-hooks, packaging, importlib-metadata, unearth, requests-toolbelt, findpython, cachecontrol, pdm
/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Successfully installed blinker-1.5 cachecontrol-0.12.11 certifi-2022.12.7 charset-normalizer-2.1.1 commonmark-0.9.1 distlib-0.3.6 filelock-3.8.2 findpython-0.2.2 idna-3.4 importlib-metadata-5.1.0 installer-0.6.0 lockfile-0.12.2 msgpack-1.0.4 packaging-21.3 pdm-2.3.2 platformdirs-2.6.0 pygments-2.13.0 pyparsing-3.0.9 pyproject-hooks-1.0.0 python-dotenv-0.21.0 requests-2.28.1 requests-toolbelt-0.10.1 resolvelib-0.9.0 rich-12.6.0 shellingham-1.5.0 tomli-2.0.1 tomlkit-0.11.6 unearth-0.6.3 urllib3-1.26.13 virtualenv-20.17.1 zipp-3.11.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

The pdm update step:

$ pdm update -v $EXTRA_PDM_UPDATE_ARGS --no-sync
python.use_venv is on, creating a virtualenv for this project...
Run command: ['/usr/local/bin/python', '-m', 'virtualenv', '/builds/software/py-beta-data-warehouse/.venv', '-p', '/usr/local/bin/python', '--no-pip', '--no-setuptools', '--no-wheel', '--prompt=py-beta-data-warehouse-3.9']
created virtual environment CPython3.9.16.final.0-64 in 117ms
  creator CPython3Posix(dest=/builds/software/py-beta-data-warehouse/.venv, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Virtualenv is created successfully at /builds/software/py-beta-data-warehouse/.venv
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   apache-beam[gcp]>=2.22.0
pdm.termui:   tzlocal
pdm.termui:   google-cloud-datastore~=1.7
pdm.termui:   google-cloud-storage==1.29.0
pdm.termui:   google-cloud-bigquery==1.24.0
pdm.termui:   google-api-python-client==2.13.0
pdm.termui:   google-api-core==1.33.2
pdm.termui:   google-cloud-pubsub==2.9.0
pdm.termui:   google-cloud-logging>=2.2.0
pdm.termui:   google-auth==1.35.0
pdm.termui:   pandas<1.5.2,>=1.3.5
pdm.termui:   toml
pdm.termui:   py-beta-data-utils>=0.5.0
pdm.termui:   py-beta-common[cloud]>=1.7.2
pdm.termui:   click>=8.0.3
pdm.termui:   pyarrow>=7.0.0
pdm.termui:   mkdocs>=1.3.1
pdm.termui:   mkdocs-material>=8.3.9
pdm.termui:   mkdocstrings[python]>=0.19.0
pdm.termui:   mkdocs-gen-files>=0.3.5
pdm.termui:   mkdocs-literate-nav>=0.4.1
pdm.termui:   mkdocs-section-index>=0.3.4
pdm.termui:   mkdocs-include-markdown-plugin>=3.6.1
pdm.termui:   mkdocs-click>=0.8.0
pdm.termui:   flake8>=4.0.1
pdm.termui:   pep8-naming>=0.12.1
pdm.termui:   flake8-docstrings>=1.6.0
pdm.termui:   pytest>=7.1.1
pdm.termui:   pytest-cov>=3.0.0
pdm.termui:   pytest-xdist>=2.5.0
pdm.termui:   python>=3.8,<3.12
pdm.termui:   Adding requirement apache-beam[gcp]>=2.22.0
pdm.termui:   Adding requirement tzlocal
pdm.termui:   Adding requirement google-cloud-datastore~=1.7
pdm.termui:   Adding requirement google-cloud-storage==1.29.0
pdm.termui:   Adding requirement google-cloud-bigquery==1.24.0
pdm.termui:   Adding requirement google-api-python-client==2.13.0
pdm.termui:   Adding requirement google-api-core==1.33.2
pdm.termui:   Adding requirement google-cloud-pubsub==2.9.0
pdm.termui:   Adding requirement google-cloud-logging>=2.2.0
pdm.termui:   Adding requirement google-auth==1.35.0
pdm.termui:   Adding requirement pandas<1.5.2,>=1.3.5
pdm.termui:   Adding requirement toml
pdm.termui:   Adding requirement py-beta-data-utils>=0.5.0
pdm.termui:   Adding requirement py-beta-common[cloud]>=1.7.2
pdm.termui:   Adding requirement click>=8.0.3
pdm.termui:   Adding requirement pyarrow>=7.0.0
pdm.termui:   Adding requirement mkdocs>=1.3.1
pdm.termui:   Adding requirement mkdocs-material>=8.3.9
pdm.termui:   Adding requirement mkdocstrings[python]>=0.19.0
pdm.termui:   Adding requirement mkdocs-gen-files>=0.3.5
pdm.termui:   Adding requirement mkdocs-literate-nav>=0.4.1
pdm.termui:   Adding requirement mkdocs-section-index>=0.3.4
pdm.termui:   Adding requirement mkdocs-include-markdown-plugin>=3.6.1
pdm.termui:   Adding requirement mkdocs-click>=0.8.0
pdm.termui:   Adding requirement flake8>=4.0.1
pdm.termui:   Adding requirement pep8-naming>=0.12.1
pdm.termui:   Adding requirement flake8-docstrings>=1.6.0
pdm.termui:   Adding requirement pytest>=7.1.1
pdm.termui:   Adding requirement pytest-cov>=3.0.0
pdm.termui:   Adding requirement pytest-xdist>=2.5.0
pdm.termui:   Adding requirement python>=3.8,<3.12
pdm.termui: ======== Starting round 0 ========
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
unearth: Downloading <Link https://files.pythonhosted.org/packages/09/d4/8f1e3f5e6650da1686901306ca560d3290367e9fa38b3ff9da3b05e97d4b/google_api_core-1.33.2-py3-none-any.whl (from None)> (115 kB)
pdm.termui:   Adding requirement googleapis-common-protos<2.0dev,>=1.56.2(from google-api-core 1.33.2)
pdm.termui:   Adding requirement protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<4.0.0dev,>=3.19.5(from google-api-core 1.33.2)
pdm.termui:   Adding requirement google-auth<3.0dev,>=1.25.0(from google-api-core 1.33.2)
pdm.termui:   Adding requirement requests<3.0.0dev,>=2.18.0(from google-api-core 1.33.2)
pdm.termui: Pinning: google-api-core 1.33.2
pdm.termui: ======== Ending round 1 ========
pdm.termui: ======== Starting round 2 ========
unearth: Downloading <Link https://files.pythonhosted.org/packages/fb/7a/1b3eb54caee1b8c73c2c3645f78a382eca4805a301a30c64a078e736e446/google_auth-1.35.0-py2.py3-none-any.whl (from None)> (152 kB)
pdm.termui:   Adding requirement cachetools<5.0,>=2.0.0(from google-auth 1.35.0)
pdm.termui:   Adding requirement pyasn1-modules>=0.2.1(from google-auth 1.35.0)
pdm.termui:   Adding requirement setuptools>=40.3.0(from google-auth 1.35.0)
pdm.termui:   Adding requirement six>=1.9.0(from google-auth 1.35.0)
pdm.termui:   Adding requirement rsa<5,>=3.1.4; python_version >= "3.6"(from google-auth 1.35.0)
pdm.termui: Pinning: google-auth 1.35.0
pdm.termui: ======== Ending round 2 ========
pdm.termui: ======== Starting round 3 ========
unearth: Downloading <Link https://files.pythonhosted.org/packages/76/e2/84e402f4258fac30c870607210dd0931d7c8ad522e9a6871ad6641151863/google_api_python_client-2.13.0-py2.py3-none-any.whl (from None)> (7.1 MB)
pdm.termui:   Adding requirement httplib2<1dev,>=0.15.0(from google-api-python-client 2.13.0)
pdm.termui:   Adding requirement google-auth<2dev,>=1.16.0(from google-api-python-client 2.13.0)
pdm.termui:   Adding requirement google-auth-httplib2>=0.1.0(from google-api-python-client 2.13.0)
pdm.termui:   Adding requirement google-api-core<2dev,>=1.21.0(from google-api-python-client 2.13.0)
pdm.termui:   Adding requirement six<2dev,>=1.13.0(from google-api-python-client 2.13.0)
pdm.termui:   Adding requirement uritemplate<4dev,>=3.0.0(from google-api-python-client 2.13.0)
pdm.termui: Pinning: google-api-python-client 2.13.0
pdm.termui: ======== Ending round 3 ========

... snip ...

lambda added the 🐛 bug Something isn't working label Dec 8, 2022

lambda mentioned this issue Dec 8, 2022

Old google-api-python-client versions are not PEP 440 compliant pypa/packaging#627

Closed

frostming mentioned this issue Dec 8, 2022

fix: skip invalid versions rejected by packaging 22.0 frostming/unearth#26

Merged

frostming closed this as completed in frostming/unearth#26 Dec 8, 2022

frostming mentioned this issue Dec 8, 2022

chore: update deps unearth and packaging #1555

Merged

2 tasks

lambda mentioned this issue Dec 8, 2022

Pin version of packaging and other essential dependencies in PDM #1558

Closed

tintinthong mentioned this issue Dec 11, 2022

Fix pdm install. New requirement parser giving trouble cardstack/cardstack#3489

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PDM fails to resolve on packages which have old versions with legacy version identifiers #1554

PDM fails to resolve on packages which have old versions with legacy version identifiers #1554

lambda commented Dec 8, 2022

frostming commented Dec 8, 2022

frostming commented Dec 8, 2022

pradyunsg commented Dec 8, 2022

lambda commented Dec 8, 2022

PDM fails to resolve on packages which have old versions with legacy version identifiers #1554

PDM fails to resolve on packages which have old versions with legacy version identifiers #1554

Comments

lambda commented Dec 8, 2022

Steps to reproduce

Actual behavior

Expected behavior

Workaround

Further information

frostming commented Dec 8, 2022

frostming commented Dec 8, 2022

pradyunsg commented Dec 8, 2022

lambda commented Dec 8, 2022