creating venv with specific --python version (not in PATH variable) fails after 2023.4.29

[mjhalwa]

Issue description

what fails:

If I try to setup a venv using pipenv install --python 3.11 or pipenv install on a directory that contains a Pipfile with [requires] Python = "3.11" e.g.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.11"
python_full_version = "3.11.4"

pipenv fails with the following error:

Warning: Python 3.11 was not found on your system...
Neither 'pyenv' nor 'asdf' could be found to install Python.
You can specify specific versions of Python with:
$ pipenv --python path\to\python

(same happens for other Python Versions like 3.10)

what works

If instead running pipenv install on a plain directory or pipenv install on a directory with following Pipfile (without [requires])

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

it creates a venv with my latest installed Python version 3.11.4

Expected result

create a venv with python from my installation of Python 3.11 at C:\Software\Python\Python3_11_4

Actual result

pipenv install --python 3.11 --verbose returns

Using python: 3.11
Path to python: None
Warning: Python 3.11 was not found on your system...
Neither 'pyenv' nor 'asdf' could be found to install Python.
You can specify specific versions of Python with:
$ pipenv --python path\to\python

Steps to replicate

I am running on Windows 10 64bit.

1. install Python 3.11.4 in C:\Software\Python\Python3_11_4

2. add C:\Software\Python\Python3_11_4\Scripts to System PATH

3. open powershell (v7) and run py -3.11 -m pip install pipenv (which installs latest pipenv 2023.8.28)

4. go to empty directory and run pipenv install --python 3.11 -> ERROR

5. but go to empty directory and run pipenv install works and creates following Pipfile (without [requires]):

   [[source]]
   url = "https://pypi.org/simple"
   verify_ssl = true
   name = "pypi"
   
   [packages]
   
   [dev-packages]

6. also go to empty directory and run pipenv install --python C:\Software\Python\Python3_11_4 works

7. uninstall pipenv py -3.11 -m pip uninstall pipenv

8. install older pipenv<=2023.4.29 py -3.11 -m pip install pipenv==2023.4.29

9. go to empty directory and run pipenv install --python 3.11 -> works and creates following Pipfile (with [requires])

   [[source]]
   url = "https://pypi.org/simple"
   verify_ssl = true
   name = "pypi"
   
   [packages]
   
   [dev-packages]
   
   [requires]
   python_version = "3.11"
   python_full_version = "3.11.4"

all newer versions than 2023.4.29 fail installing a venv with specific python version and, all up to 2023.4.29 seem to work (I tested 2022.6.7, 2022.11.11, 2023.2.4 and 2023.4.29). Testing with 2023.5.19, 2023.6.2, 2023.7.1 and latest 2023.8.28 all failed.

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2023.8.28'

Pipenv location: 'C:\\Software\\Python\\Python3_11_4\\Lib\\site-packages\\pipenv'

Python location: 'C:\\Software\\Python\\Python3_11_4\\python.exe'

OS Name: 'nt'

User pip version: '23.2.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.4',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.19045',
 'python_full_version': '3.11.4',
 'python_version': '3.11',
 'sys_platform': 'win32'}

...

I omitted PATH and environment variables and hope they are currently not relevant.

[matteius]

@mjhalwa Unfortunately I don't see the system PATH in the support info, so I only have to go by that you added it to the path -- it should find it, I haven't personally run into this issue.

[mjhalwa]

Sorry for the delay. I was able to investigate some more.

My PATH (reduced to python-related stuff) contains following paths in order:

• C:\Software\Python\Python3_11_4\Scripts
• C:\Users\SYSTEM\AppData\Local\Programs\Python\Launcher
• ...
• C:\WINDOWS
• ...
• C:\Users\%USERNAME%\AppData\Local\Programs\Python\Launcher
• ...

py --list shows me I have following python versions installed on my machine:

 -V:3.11 *        Python 3.11 (64-bit)
 -V:3.10          Python 3.10 (64-bit)
 -V:3.9           Python 3.9 (64-bit)
 -V:3.7           Python 3.7 (64-bit)
 -V:2.7           Python 2.7

Note: all are installed in separated paths next to Python3_11_4 and cann be invoked with py and minor version e.g. py -3.7

From comparing to another machine where pipenv seemed to work, I figured out I was missing C:\Software\Python\Python3_11_4\ - I only had the C:\Software\Python\Python3_11_4\Scripts

Now when adding C:\Software\Python\Python3_11_4\ after the ...\Scripts the situation is

• with pipenv versions 2023.9.8 as well as 2023-10-3:
  • pipenv install creates Pipfile containing the privious missing python_version = "3.11" line
  • pipenv install --python 3.11 now works and creates a valid Pipfile with python_version = "3.11" line
  • BUT with all other versions than 3.11 (i checked all my installed versions), say e.g. pipenv install --python 3.10 I again get the error mentioned in my initial report

    Warning: Python 3.10 was not found on your system...
    Neither 'pyenv' nor 'asdf' could be found to install Python
    ...
    

• still with pipenv==2023.4.29 (only uninstalling pipenv and reinstalling pipenv==2023.4.29 with same PATH etc.) everything works, including all my installed Python versions.

So I see the pattern, that for pipenv>2023.4.29 only python versions work, that are explicitly written to my PATH with both the bin directory and the Scripts subdirectory. I verified for my python 3.9 by adding it to the PATH that is now (only python related paths):

• C:\Software\Python\Python3_11_4\Scripts
• C:\Software\Python\Python3_11_4\
• C:\Software\Python\Python3_9_5\Scripts
• C:\Software\Python\Python3_9_5\
• C:\Users\SYSTEM\AppData\Local\Programs\Python\Launcher
• ...
• C:\WINDOWS
• ...
• C:\Users\%USERNAME%\AppData\Local\Programs\Python\Launcher
• ...

And indeed pipenv install --python 3.9 now works, while still pipenv install -python 3.10 does not work.

Do I need to carry all installed Python Versions in the PATH to be able to invoke them with pipenv? Is this behavior somehow expected in newer versions or might there still be something missing on my machine? - like is it expected to have Python installed in AppData and it does not work because I install to a custom directory - I expected it will work, because py is able to find all python versions.

[mjhalwa]

I tried on a fresh windows 10 now to

1. install Python 3.12 with "Custom Installation", checking "add python to PATH" and "install py launcher with Admin privileges"

   Note: my PATH (echo $Env:PATH) now contains ...;C:\WINDOWS\system32;C:\WINDOWS;...;C:\Software\Python\Python_3_12 \Scripts\;C:\Software\Python\Python_3_12\;...

2. install pipenv with py -3.12 -m pip install pipenv

   Note: this installed pipenv, version 2023.10.3

3. in a test directory run pipenv install --python 3.12 works as expected

4. remove venv with pipenv --rm then delete Pipfile and Pipfile.lock

5. remove just C:\Software\Python\Python_3_12\; from PATH
   6 retry pipenv install --python 3.12 -> Error

6. retry pipenv install -> creates venv, but without Python Version tag in Pipfile (as described in my original post)

Thus it seems to me that pipenv does only recognize Python Versions where the binary can be found in the PATH.

Additonally I installed a Python 3.10.10 with leaving the Python installer at is defaults and "Install Now" (leaving "add python to PATH" unchecked) and without "Disable path length limit"

• I can confirm there is not specific Python 3.10.10 in my PATH
• python does not work (as I left the 3.12.0` removed)
• py -3.12 starts a python 3.12.0 shell
• py -3.10 starts a python 3.10.10 shell
• pipenv install --python 3.10 stops with the explained error

Thus, I guess, I can confirm, pipenv does also not work with standard-python installations if they are not added to the PATH.

If I now add C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python310 to the PATH, then pipenv install --python 3.10 works (on a fresh powershell window)

Some additional checks:

• same behavior on Windows Powershell (version 5.2) and CMD

• without Python Installation directories in path following commands create equivalent errors, similar to the reported Error as they cannot find the requested python version:

  pipenv install --python 3.10
  pipenv install --python 3.12
  py -3.12 -m pipenv install --python 3.10
  py -3.12 -m pipenv install --python 3.12
  

Now, uninstalling the latest pipenv and force install pipenv==2023.4.29, suddenly without any further change all above commands simply work and find all python versions.

@matteius I hope you can now reproduce my issue

For me current releases do not meet the expected behavior, as having old python versions in my PATH makes no sense and even the latest python version is not required in my PATH as Py-Launcher detects all python versions successfully. I would expect pipenv to either work like py or query py for a list of python install directory (e.g. py --list-paths), thereby also the py-registered default Python version could be in sync. - especially as versions up to 2023.4.29 were already able to pick up all installed python versions (even if not defined in the PATH)

[ericPrince]

I am seeing the same behavior on Windows with Pyenv. I used to be able to pipenv install ... and have it use pyenv to pick the correct Python version listed either in [requires] or in --python X.Y, but it no longer works and I get Warning: Python X.Y was not found on your system.... Tested on both Powershell and Bash for Windows. I tried updating pyenv to the latest version, but that did not help.

Note that I see the same behavior where older versions of pipenv used to find and use the correct Python version, but newer pipenv releases aren't doing this on Windows.

I'm also seeing a related Stack Overflow question: https://stackoverflow.com/questions/76478886/version-2023-5-19-or-later-pipenv-fails-to-recognize-various-pythons-installed

[mjhalwa]

just tried latest pipenv-2023.12.1 and pipenv install --python 3.11 still does not find my installed python 3.11 if not listed in PATH variable

[mjhalwa]

I was thinking of a way to contribute with a PR to fix this. While investigating where this problem arises in pipenv, I figured out that the problem actually goes down to the dependency pythonfinder that was updated to version 2.0.0 in Release 2023.5.19. pythonfinder still claims to supports windows via both the PATH environment variable and PEP-514 compliant finder.

So I tried installing pythonfinder globally to run

from pythonfinder import Finder

f = Finder()
f.find_python_version(3, minor=11)

f = Finder(global_search=True)
f.find_python_version(3, minor=11)

f = Finder(system=True)
f.find_python_version(3, minor=11)

f = Finder(global_search=True, system=True)
f.find_python_version(3, minor=11)

, but all of these only returned my installation of Python if it was part of the Windows PATH. So the PATH seems to work, but the PEP-514 part seems actually to be broken, what is discussed in pythonfinder Issue 158.

@matteius This way I learned about your PR to pipenv #6140 that would actually fix this issue (if I understand correctly). Thank you for working on it, please let me know if I could support somehow (e.g. testing a possible fix on my machine)

[EDIT] In case this helps, I checked the possible registry keys according to PEP-514 that could store the Python installation and found my 3 installations of python only under HKEY_CURRENT_USER with company PythonCore:

• HKEY_CURRENT_USER\Software\Python\PythonCore\3.10\InstallPath
• HKEY_CURRENT_USER\Software\Python\PythonCore\3.11\InstallPath
• HKEY_CURRENT_USER\Software\Python\PythonCore\3.12\InstallPath

I do not have an entry HKEY_LOCAL_MACHINE\Software\Python and only have PyLauncher at HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Python\PyLauncher

[matteius]

@mjhalwa if you could take a look again at the pipenv pythonfinder PR -- I updated it tonight. I am not sure if it works 100% but I do want to get this issue patched, even if its not 100% perfect, if its a functional improvement is what I am going for this iteration before shipping pipenv 3000 .... willing to make it 100% perfect though if we get enough eyes on it and suggestions to get it there.

[mjhalwa]

Currently I just have C:\Software\Python\Python_3_12\Scripts in Path. Not the Python_3_12 nor any other Python paths and I can call ´py-Launcher directly from console. py`-Launcher finds following paths and can invoke both pythons:

py --list-paths
 -V:3.12 *        C:\Software\Python\Python_3_12\python.exe
 -V:3.11          C:\Software\Python\Python_3_11\python.exe
 -V:3.10          C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python310\python.exe

I have following python installation pats in regedit on this machine.

• Computer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.10\InstallPath
• Computer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.11\InstallPath
• Computer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.12\InstallPath

Note: that on this machine SOFTWARE is all upper case letters instead of Software that I posted last time on a different machine (don't know if this makes some difference)

So I cloned the branch of your PR #6140 on Commit a26d677 . I ran

cd pipenv
git checkout pythonfinder-pep514
pipenv sync --dev
pipenv shell

Note: this creates a Python 3.12 venv.

and then navigated in the shell environment to some empty directory and tried (in between always running pipenv --rm and deleting created Pipfile and Pipfile.lock)

Note: same results with Powershell v5.1, cmd.exe and Git Bash

• ⚡ pipenv install throws an exception

  Traceback (most recent call last):
    File "<frozen runpy>", line 198, in _run_module_as_main
    File "<frozen runpy>", line 88, in _run_code
    File "C:\Users\%USERNAME%\.virtualenvs\pipenv-xD7V_B86\Scripts\pipenv.exe\__main__.py", line 7, in <module>
      # when invoked as python -m pip <command>
                  ^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\core.py", line 1157, in __call__
      return self.main(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\cli\options.py", line 58, in main
      return super().main(*args, **kwargs, windows_expand_args=False)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\core.py", line 1078, in main
      rv = self.invoke(ctx)
          ^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\core.py", line 1688, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\core.py", line 1434, in invoke
      return ctx.invoke(self.callback, **ctx.params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\core.py", line 783, in invoke
      return __callback(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\decorators.py", line 92, in new_func
      return ctx.invoke(f, obj, *args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\click\core.py", line 783, in invoke
      return __callback(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\cli\command.py", line 209, in install
      do_install(
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\routines\install.py", line 60, in do_install
      ensure_project(
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\utils\project.py", line 48, in ensure_project
      ensure_virtualenv(
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\utils\virtualenv.py", line 149, in ensure_virtualenv
      python = ensure_python(project, python=python)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\utils\virtualenv.py", line 242, in ensure_python
      path_to_python = find_a_system_python(python)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\utils\virtualenv.py", line 352, in find_a_system_python
      return next(iter(finder.find_all_python_versions()), None)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\pythonfinder.py", line 198, in find_all_python_versions
      versions = self.system_path.find_all_python_versions(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\models\path.py", line 447, in find_all_python_versions
      values = list(self.get_pythons(sub_finder))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\models\path.py", line 421, in get_pythons
      pythons = [entry for entry in self._get_all_pythons(finder)]
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\models\path.py", line 413, in _get_all_pythons
      for python in self._filter_paths(finder):
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\models\path.py", line 408, in _filter_paths
      python_version = finder(path)
                      ^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\models\path.py", line 437, in sub_finder
      return obj.find_all_python_versions(major, minor, patch, pre, dev, arch, name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\%USERNAME%\Desktop\Test\pipenv\pipenv\vendor\pythonfinder\models\python.py", line 258, in find_all_python_versions
      next(iter(py for py in base.find_all_python_versions()), None)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'WindowsLauncherEntry' object has no attribute 'find_all_python_versions'
  

• ✅ pipenv install --python 3.12 works creates following Pipfile:

  [[source]]
  url = "https://pypi.org/simple"
  verify_ssl = true
  name = "pypi"
  
  [packages]
  
  [dev-packages]
  
  [requires]
  python_version = "3.12"
  python_full_version = "3.12.0"
  

• ❌ pipenv install --python 3.11 fails with

  Warning: Python 3.11 was not found on your system...
  Neither 'pyenv' nor 'asdf' could be found to install Python.
  You can specify specific versions of Python with:
  $ pipenv --python path\to\python
  

• ❌ pipenv install --python 3.10 fails with

  Warning: Python 3.10 was not found on your system...
  Neither 'pyenv' nor 'asdf' could be found to install Python.
  You can specify specific versions of Python with:
  $ pipenv --python path\to\python
  

In comparison with the latest official Version 2023.12.1 (installed globally via py -m pip install pipenv):

• ✅ pipenv install works and creates a Pipfile without the last line python_full_version = "3.12.0"
• ❌ pipenv install --python 3.12 already fails with Warning: Python 3.12 was not found on your system...
• ❌ pipenv install --python 3.11 fails as well with Warning: Python 3.11 was not found on your system...
• ❌ pipenv install --python 3.10 fails as well with Warning: Python 3.10 was not found on your system...

and also tested main on Commit 6abb08c:

• ✅ pipenv install works and creates a Pipfile without the last line python_full_version = "3.12.0"
• ✅ pipenv install --python 3.12 works too with additional line python_full_version = "3.12.0" in Pipfile (as in your PR)
• ❌ pipenv install --python 3.11 fails as well with Warning: Python 3.11 was not found on your system...
• ❌ pipenv install --python 3.10 fails as well with Warning: Python 3.10 was not found on your system...

Compared to my original Issue:

• both the latest official version 2023.12.1 and latest Commit 6abb08c on main already add the entirely missing [requires] block to the Pipfile on pipenv install
• your PR and the latest Commit on main have the slight improvement that they support pipenv install --python 3.12 with the virtual environment's python install (wrong interpretation), actually this just uses the virtual environment's Python as the line Using C:/Users/%USERNAME%/.virtualenvs/pipenv-xD7V_B86/Scripts/python.exe (3.11.9) to create virtualenv... indicates. This can easily be shown by adding lines [requires] and python_version = "3.11" to the pipenv project Pipfile and recreating this virtual environment, suddenly pipenv install --python 3.11 works and non-venv-versions pipenv install --python 3.12 and pipenv install --python 3.10 fail

I keep this machine just in case you have any more improvements to test. @matteius How do you feel about adding a flag to list all python installations that pipenv can find like py-Launcher has on py --list-paths. Probably this would simplify probing found python installations and could be handy anyways? How about querying python installations from py-Launcher (if installed)?

Hope this helps a little.

[mjhalwa]

@matteius I was wondering if chaging all occurrences of Software\ to SOFTWARE\ in the source of 6abb08c might resolve my problems to better reflect the regedit path on my machine, but the results are unchanged.

I changed following strings

• in patched/pip/_vendor/platformdirs/windows.py Line 199
• in patched/pip/_vendor/pygments/formatters/img.py Line 168 - Line 171
• in patched/pip/_vendor/requests/utils.py Line 85
• in utils/funktools.py Line 199
• in vendor/pythonfinder/models/python.py Line 376

I also noticed the reason why Version 3.12 works above is due to the python interpreter in the venv (confirmed with python --version inside the pipenv shell). Adding

[requires]
python_version = "3.11"

to the Pipfile of 6abb08c , made pipenv install --python 3.12 fail, but pipenv install --python 3.11 work instead.