Cannot use custom pypi.url to add dependencies for some private pypi

[monchin]

My company use a private pypi.url, and it works well when I

pip config set global.index-url http://10.1.10.100:8084/root/custom
pip config set global.trusted-host 10.1.10.100

Now I want to use pdm, and I set

pdm config --local pypi.url http://10.1.10.100:8084/root/custom
pdm config --local pypi.verify_ssl False

and check configs by

PS C:\Project\pdm-prj\test_app> pdm config pypi.url
http://10.2.40.16:8084/root/custom
PS C:\Project\pdm-prj\test_app> pdm config pypi.verify_ssl
False

now I want to add numpy==1.26.1, so I

pdm add numpy==1.26.1

and error occurs

Adding packages to default dependencies: numpy==1.26.1
See C:\Users\c.men\AppData\Local\Temp\pdm-lock-jitqaql4.log for detailed debug log.
[CandidateNotFound]: Unable to find candidates for numpy. There may exist some issues with the package name or network
condition.
Add '-v' to see the detailed traceback

I checked the log, finding that pdm could not analyse the address correctly

pdm.termui: ======== Start resolving requirements ========
pdm.termui:   numpy==1.26.1
pdm.termui:   python>=3.10,<3.11
pdm.termui:   Adding requirement numpy==1.26.1
unearth.collector: Collecting links from http://10.1.10.100:8084/root/custom/numpy/
unearth.auth: Found index url http://10.1.10.100:8084/root/custom/
unearth.collector: Fetching HTML page http://10.1.10.100:8084/root/custom/numpy/
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/ (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: 
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/+searchhelp (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: +searchhelp
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/custom/numpy (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: numpy
unearth.evaluator: Skipping link <Link https://pypi.org/pypi?%3Aaction=list_classifiers (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: pypi
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/ (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: 
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: root
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/custom (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: custom
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/custom/numpy (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: numpy
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/+status (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: +status
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/custom/+simple/numpy (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: numpy
unearth.evaluator: Skipping link <Link https://mirrors.aliyun.com/pypi/simple/numpy/ (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: 
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/pypi (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: pypi
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/pypi/numpy/1.26.1 (from http://10.1.10.100:8084/root/custom/numpy)>: Unsupported archive format: 1.26.1
unearth.evaluator: Skipping link <Link http://10.1.10.100:8084/root/pypi (from http://10.1.10.100:8084/root/custom/numpy)>: Not a file: pypi
...

Please look at the second last line above "...", it said Skipping link <Link http://10.1.10.100:8084/root/pypi/numpy/1.26.1 , so I checked http://10.1.10.100:8084/root/pypi/numpy, and found it a little different from https://pypi.org/simple/numpy, that is, there are directly different .whl numpy files in https://pypi.org/simple/numpy/, but only versions in my company's http://10.1.10.100:8084/root/pypi/numpy/, and .whl files are in the next level, e.g., http://10.1.10.100:8084/root/pypi/numpy/1.26.1, but pdm cannot resolve the next level, so the error occurs.

I guess for more universal use, pdm should change code in unearth.colloctor.IndexHTMLParser.parse_html_page. I'm not familiar with html, but I guess perhaps a url should be judged whether it is a folder or a file, if it is a folder then the next level should be searched, and the right .whl file could then be found.

[pawamoy]

Is your custom index compliant with PEP 503? https://packaging.python.org/en/latest/specifications/simple-repository-api/

[pawamoy]

No worries! I can't guarantee it, but I suspect unearth only works with PEP 503 compliant indices, and will not implement any kind of support for non-compliant ones. For what it's worth, I personally use pypiserver to self-host a private index. It works fine with PDM.

[frostming]

I roughly searched the code of pip and don't think it can recursively retrieve package links. Even if pip supports it, PDM won't do because it doesn't comply with PEP 503.

[monchin]

Hello, I asked our company's administrator, and he said that he used devpi to automatically build our mirror. And if I use http://10.1.10.100:8084/root/pypi/+simple/numpy instead of http://10.1.10.100:8084/root/pypi/numpy, then all the .whl files would be there, just like https://pypi.org/simple/numpy. but I still don't know why there is +simple but not simple like pypi.org, maybe I need to know how devpi works.

[frostming]

Then just assigning the index url to http://10.1.10.100:8084/root/pypi/+simple/ would work

[monchin]

Thank you, it works!