Failling with custom versioning for python package.

[MrLixm]

Hello, my issue is related to my workflow so I don't except much, but reporting in case it can be of any use.

traceback

 ./rez-pip2.exe progress
INFO     Installing requested packages for Python 3.9.13.1

Usage:
  pip.pyz install [options] <requirement specifier> [package-index-options] ...
  pip.pyz install [options] [-e] <vcs project url> ...
  pip.pyz install [options] [-e] <local project path> ...
  pip.pyz install [options] <archive url/path> ...

--python-version error: invalid --python-version value: '3.9.13.1': at
rez_pip\\data\\pip.pyz install -q progress --disable-pip-version-check --dry-run --ignore-installed --python-version=3.9.13.1 --only-binary=:all: --target=/tmp/asd --disable-pip-version-check --report C:\\Users\\lcoll\\AppData\\Local\\Temp\\pip-install-outputb2h27l70'

reason

I am using a custom versioning schema for all vendor packages by appending an additional version token. Example a python-3.9.13 can become python-3.9.13.1.
It seems that rez-pip has no issue by founding my python version, though pip of course doesn't like the non-standard version and fail.

potential-fix

1. I'm not sure it would be desirable to include a python version "sanitize" that would strip out any non standard token ?
2. Or perhaps it could read the python version from a custom package attribute instead ?
3. I think the best solution would just to let the use provide the actual version to pip, i.e

   rez-pip2 whatever --python-version 3.9.13.1 -- --python-version 3.9.13
   

   which currently doesn't work as pip args are append to the existing pip command and doesn't override the existing args

context

• windows 10
• rez-pip 0.2.0 from pip
• rez 2.113.0

[JeanChristopheMorinPerso]

Hi @MrLixm, thanks for opening this issue! It's an interesting us case that I haven't put a lot of thought into. I started to think about the package name but not the version.

I was thinking of potentially adding a way to configure the package name and allowing to use rez-pip with PyPy, Jython (and all the other python implementations). And I think it would be valid to also support custom version schemes. We could do that by adding a rez_pip_python_version setting that could support something like a regex, or a rez_pip_python_version_attr that would default to version, and could be overridden with a custom package attribute of your choice (which is your option 2 you mentioned). Your option 3 could work but I'm not fully sold yet.

I'll need some time to think about it. Or all the options I mentioned, do you prefer one over the other?

[MrLixm]

Cool to know it's in your plans.
I think I see the 2 config settings you propose more as a long-term solution. They will offer full flexibility so that sounds perfect.
As a short-term solution I think I would argue for option 3. I suggested. At least for me it make sense that if you specify a pip arg in the original command, it overrides the similar pip arg created by rez-pip (so not only --python-version). But I don't know if it's actually possible because rez-pip will know about 3.9.13.1 while pip will actually "write" 3.9.13.

[SuperValou]

Just for brainstorming, maybe we could introduce some --python-provider argument and have this behaviour:

• rez-pip2 some_pkg: attempts to use the latest version of a rez package called python. If the version number of this package makes sense for pip (e.g. "3.9.13"), use it for pip; otherwise (e.g. "3.9.13.1"), raise an error

• rez-pip2 some_pkg --python-provider my_custom_python: attempts to use the latest version of a rez package called my_custom_python (assuming it provides a "python" command). If the version number of this package makes sense for pip (e.g. "3.9.13"), use it for pip; otherwise (e.g. "3.9.13.1") raise an error

• rez-pip2 some_pkg --python-provider my_custom_python --python-version 3.9.13: attempts to use the latest version of a rez package called my_custom_python. Regardless of this package version, use "3.9.13" for pip.

• rez-pip2 some_pkg --python-provider my_custom_python-3.9.13.1 --python-version 3.9.13: attempts to use the 3.9.13.1 version of a rez package called my_custom_python. Regardless of this package version, use "3.9.13" for pip.

• rez-pip2 some_pkg --python-provider maya-2023 --python-version 3.9.7: attempts to use the 2023 version of a rez package called maya. Regardless of this package version, use "3.9.7" for pip.

The idea would be to "provide" the python interpreter through this --python-provider argument, and map it to a pip friendly python version through the --python-version arg.

EDIT: see the following comments as this idea is forgetting rez-pip2 loops through several python version

[MrLixm]

Agree with the above suggestion, the command looks clear and its nice to specify a python package that is not named python.

workaround

For now I have been trying to find a workaround that would allow me to use rez-pip2.

• As such I have created a python-3.9.13 package using rez cp python-3.9.13.1 --reversion 3.9.13.
• I then call rez python -m rez_pip pytest --python-version 3.9.13
• The correct package seems to be picked up properly the first time, pip is downloading properly all the dependencies
• but then it fail on a second time where I don't know why but 3.9.13.1 is being picked up again

traceback (note that I'm using python version `3.10.11` but its the same logic)

rez python -m rez_pip pytest --python-version 3.10.11
INFO     Installing requested packages for Python 3.10.11
INFO     Resolved 7 dependencies for python 3.10.11
INFO     Downloading...
INFO     Downloaded tomli-2.0.1 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\tomli-2.0.1-py
        3-none-any.whl' (12757 bytes)
INFO     Downloaded iniconfig-2.0.0 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\iniconfig-2.0.
        0-py3-none-any.whl' (5892 bytes)
INFO     Downloaded pluggy-1.3.0 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\pluggy-1.3.0-p
        y3-none-any.whl' (18934 bytes)
INFO     Downloaded colorama-0.4.6 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\colorama-0.4.6
        -py2.py3-none-any.whl' (25335 bytes)
INFO     Downloaded pytest-7.4.2 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\pytest-7.4.2-p
        y3-none-any.whl' (324497 bytes)
INFO     Downloaded packaging-23.2 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\packaging-23.2
        -py3-none-any.whl' (53011 bytes)
INFO     Downloaded exceptiongroup-1.1.3 to
        'C:\\Users\\lcoll\\AppData\\Local\\Temp\\rez-pip-targetwfoay1r9\\wheels\\exceptiongroup
        -1.1.3-py3-none-any.whl' (14710 bytes)
INFO     Downloaded 7 wheels
INFO     Installing pytest-7.4.2 wheel
INFO     Installing exceptiongroup-1.1.3 wheel
INFO     Installing pluggy-1.3.0 wheel
INFO     Installing tomli-2.0.1 wheel
INFO     Installing colorama-0.4.6 wheel
INFO     Installing iniconfig-2.0.0 wheel
INFO     Installing packaging-23.2 wheel
INFO     Creating rez package for pytest
INFO     Created 0 variants and skipped 1
INFO     Creating rez package for exceptiongroup
INFO     Created 0 variants and skipped 1
INFO     Creating rez package for pluggy
INFO     Created 0 variants and skipped 1
INFO     Creating rez package for tomli
INFO     Created 0 variants and skipped 1
INFO     Creating rez package for colorama
INFO     Created 0 variants and skipped 1
INFO     Creating rez package for iniconfig
INFO     Created 0 variants and skipped 1
INFO     Creating rez package for packaging
INFO     Created 0 variants and skipped 1
INFO     Installing requested packages for Python 3.10.11.1

Usage:
  pip.pyz install [options] <requirement specifier> [package-index-options] ...
  pip.pyz install [options] -r <requirements file> [package-index-options] ...
  pip.pyz install [options] [-e] <vcs project url> ...
  pip.pyz install [options] [-e] <local project path> ...
  pip.pyz install [options] <archive url/path> ...

--python-version error: invalid --python-version value: '3.10.11.1':
at most three version parts are allowed
rez_pip.exceptions.PipError: Failed to run pip command: 'z:\\rez\\packages\\python\\3.10.11.1\\platform-windows\\arch-amd64\\python.exe c:\\program files\\rez\\lib\\site-packages\\rez_pip\\data\\pip.pyz install -q pytest --disable-pip-version-check --dry-run --ignore-installed --python-version=3.10.11.1 --only-binary=:all: --target=/tmp/asd --disable-pip-version-check --report C:\\Users\\lcoll\\AppData\\Local\\Temp\\pip-install-outputvunb5mzi'

EDIT: I did not understood that rez-pip was finding all version matching 3.9.13 and using all of them. To prevent it _ just have to make the version strict like --python-version ==3.9.13

[SuperValou]

I forgot about rez-pip2 looping through several python versions as well, so the idea of a --python-provider arg is more complex than it seems. The rez-pip2 command would need to know the range of versions to loop through on the package providing python, and also know the mapping between the provider's version and the pip-friendly version.

Example: "loops through maya 2022 to 2023 packages, and map 2022.x version to 3.7.7 and 2023.x versions to 3.9.7". That's probably not trivial to implement.

[SuperValou]

Mmmh unless the package providing python has a custom attribute like you both previously mentionned: assuming all maya packages have some sort of provided_python_version = 3.7.7 for 2022.x and = 3.9.7 for 2023.x, the rez-pip2 command would "just" look like this: rez-pip2 some_pkg --python-provider maya --request 2022+<2024.

In this scenario, the --python-version argument would become an alias of this new --request arg, so it still feels natural to write more classic cases like rez-pip2 some_pkg --python-version 3.9.13 (that would be equivalent to rez-pip2 some_pkg --python-provider python --request 3.9.13). And rez-pip2 some_pkg --request ==3.9.13.1 would work as well (assuming the custom attribute is correctly set in the python package of course).