Support selecting Python version via tool.poetry.dependencies.python
#260
Labels
Comments
I've filed a feature request for this upstream: |
edmorley
added a commit
that referenced
this issue
Sep 3, 2024
The Python package manager Poetry is now supported for installing app dependencies: https://python-poetry.org To use Poetry apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). Apps must only have one package manager file (either `requirements.txt` or `poetry.lock`, but not both) otherwise the buildpack will abort the build with an error (which will help prevent some of the types of support tickets we see in the classic buildpack). Poetry is installed into a build-only layer, so is not available at run-time to reduce image size. The app dependencies are installed into a virtual environment (the same as for pip, after #257), which is on `PATH` so does not need explicit activation when using the app image. As such, use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not explicitly installed, since Poetry includes its own bundled copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). Both the Poetry and app dependencies layers are cached, however, the Poetry download/wheel cache is not cached, since using it is slower than caching the dependencies layer (for more details see the comments on `poetry_dependencies::install_dependencies`). The `poetry install --sync` command is run using `--only main` so as to only install the main dependencies group and not any other groups (such as test/dev/... groups). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups Work that will be handled later: - Support for selecting Python version via `tool.poetry.dependencies.python`: #260 - Build output and error messages polish/CX review (this will be performed when switching the buildpack to the new logging style). - More detailed user-facing docs: #11 Closes #7. GUS-W-9607867. GUS-W-9608286. GUS-W-9608295.
edmorley
added a commit
that referenced
this issue
Sep 4, 2024
The Python package manager Poetry is now supported for installing app dependencies: https://python-poetry.org To use Poetry, apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). Apps must only have one package manager file (either `requirements.txt` or `poetry.lock`, but not both) otherwise the buildpack will abort the build with an error (which will help prevent some of the types of support tickets we see in the classic buildpack with users unknowingly mixing and matching pip + Pipenv). Poetry is installed into a build-only layer (to reduce the final app image size), so is not available at run-time. The app dependencies are installed into a virtual environment (the same as for pip after #257, for the reasons described in #253), which is on `PATH` so does not need explicit activation when using the app image. As such, use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not installed (possible thanks to #258), since Poetry includes its own internal vendored copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). Both the Poetry and app dependencies layers are cached, however, the Poetry download/wheel cache is not cached, since using it is slower than caching the dependencies layer (for more details see the comments on `poetry_dependencies::install_dependencies`). The `poetry install --sync` command is run using `--only main` so as to only install the main `[tool.poetry.dependencies]` dependencies group from `pyproject.toml`, and not any of the app's other dependency groups (such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`). I've marked this `semver: major` since in the (probably unlikely) event there are any early-adopter projects using this CNB that have both a `requirements.txt` and `poetry.lock` then this change will cause them to error (until one of the files is deleted). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups Work that will be handled later: - Support for selecting Python version via `tool.poetry.dependencies.python`: #260 - Build output and error messages polish/CX review (this will be performed when switching the buildpack to the new logging style). - More detailed user-facing docs: #11 Closes #7. GUS-W-9607867. GUS-W-9608286. GUS-W-9608295.
edmorley
added a commit
that referenced
this issue
Sep 4, 2024
The Python package manager Poetry is now supported for installing app dependencies: https://python-poetry.org To use Poetry, apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). Apps must only have one package manager file (either `requirements.txt` or `poetry.lock`, but not both) otherwise the buildpack will abort the build with an error (which will help prevent some of the types of support tickets we see in the classic buildpack with users unknowingly mixing and matching pip + Pipenv). Poetry is installed into a build-only layer (to reduce the final app image size), so is not available at run-time. The app dependencies are installed into a virtual environment (the same as for pip after #257, for the reasons described in #253), which is on `PATH` so does not need explicit activation when using the app image. As such, use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not installed (possible thanks to #258), since Poetry includes its own internal vendored copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). Both the Poetry and app dependencies layers are cached, however, the Poetry download/wheel cache is not cached, since using it is slower than caching the dependencies layer (for more details see the comments on `poetry_dependencies::install_dependencies`). The `poetry install --sync` command is run using `--only main` so as to only install the main `[tool.poetry.dependencies]` dependencies group from `pyproject.toml`, and not any of the app's other dependency groups (such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`). I've marked this `semver: major` since in the (probably unlikely) event there are any early-adopter projects using this CNB that have both a `requirements.txt` and `poetry.lock` then this change will cause them to error (until one of the files is deleted). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups Work that will be handled later: - Support for selecting Python version via `tool.poetry.dependencies.python`: #260 - Build output and error messages polish/CX review (this will be performed when switching the buildpack to the new logging style). - More detailed user-facing docs: #11 Closes #7. GUS-W-9607867. GUS-W-9608286. GUS-W-9608295.
|
Presuming we pick option (4), then the
We would then reject anything else with an error message that says to use one of the above, or to add a Examples of specifiers we would then reject:
|
|
To add yet more things to think about - Poetry has just merged support for PEP-621 and the This means Poetry 2.0 (expected later this year, see python-poetry/poetry#3332 (comment) and python-poetry/poetry#9448) will support Given that:
... it's making me wonder whether we should instead double down on |
|
Given all of the above, I'm wontfixing this for now in favour of having a single way to specify the Python version - using a |
edmorley
added a commit
to heroku/heroku-buildpack-python
that referenced
this issue
Nov 6, 2024
After many refactoring/preparation PRs, we're now ready to add support for the package manager Poetry: https://python-poetry.org To use Poetry, apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). For now, if a `requirements.txt` or `Pipfile` is found it will take precedence over `poetry.lock` for backwards compatibility (in the future this will become a warning then an error). This means users of the third-party `python-poetry-buildpack` will need to remove that buildpack in order to use the new native Poetry support, since it exports a `requirements.txt` file during the build. Poetry is installed into the build cache rather than the slug, so is not available at run-time (since it's not typically needed at run-time and doing so reduces the slug size). The entrypoints of installed dependencies are available on `PATH`, so use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not installed since Poetry includes its own internal vendored copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). During normal (non-CI) builds, the `poetry install --sync` command is run using `--only main` so as to only install the main `[tool.poetry.dependencies]` dependencies group from `pyproject.toml` and not any of the app's other dependency groups (such as test/dev/... groups, eg `[tool.poetry.group.test.dependencies]`). On Heroku CI, all default Poetry dependency groups are installed (i.e. all groups minus those marked as `optional = true`). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups See also the Python CNB equivalent of this PR: - heroku/buildpacks-python#261 Note: We don't support controlling the Python version via Poetry's `tool.poetry.dependencies.python` field, since that field typically contains a version range, which is not safe to use. Use the newly added `.python-version` file support instead. For more on this, see the longer explanation over in the Python CNB repo: heroku/buildpacks-python#260 Closes #796. Closes #835. GUS-W-16810914.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Initial support for Poetry was added in #7, and included support for bootstrapping Poetry and then using it to install app dependencies.
This issue is for adding support for controlling the Python version via the
tool.poetry.dependencies.pythonTOML field inpyproject.tomlwhen using Poetry. This will be in addition to the existing ability to useruntime.txt, and the planned ability to use.python-version(see #6).An example Poetry config created using
poetry initwith when using Poetry 1.8.3 and Python 3.11 contains:However, the next release of Poetry is due to change the default constraint marker from
^to>=(see python-poetry/poetry#9558), which will give:It's worth noting that:
pyproject.toml(including theproject.requires-pythonfield, which is whatuvuses), which makes things more complicated both for the buildpack and for user UX (for example both syntaxes have a specifier that includes the tilde character, but they do drastically different things for some edge cases). See: https://python-poetry.org/docs/dependency-specification/poetry inituses the version of the local Python version to set the minimum Python version.pythonfield isn't specified at all, Poetry defaults to a wide range (currently>=2.7,<2.8 || >=3.4), which then breaks installing any packages that have a smaller compatibility range than that (ie: any package that only supports Python 3) - since Poetry treats itspythonfield as a "this project must be compatible with all of these Python versions" value. As such, omittingpythonreally isn't viable for users - so it's going to be set most of the time."^3.11"and">=3.11"forms allow higher major versions of Python such as 3.12 or 3.13, which can include breaking changes. Whilst supporting a range of versions makes sense for a library, for applications these unbounded ranges can cause issues (as we've seen with the Node.js buildpacks over the years), and so ideally we wouldn't want apps to use these forms.All of the above means that adding support for
tool.poetry.dependencies.pythonis going to mean making compromises in one area or another sadly, since we either have to:poetry init. Cons: Breaking changes when new major Python versions are released + encourages environment drift between CNB and local development environments - and even from one developer's machine to another.)tool.poetry.dependencies.pythonfield (either completely, or perhaps only if it uses an unsafe range) and instead install the buildpack's curated default Python version. (Pros/cons: Pretty much the same as (1), apart from the curated Python version perhaps being marginally more compatible with packages in the wild, since we wait a couple of months before making new Python versions the default.)poetry initand avoids breakage when new Python versions are released. Cons: Doesn't prevent environment drift - developers can still be using a different Python version locally - particularly as new versions are released, unless they remember to bump the project's minimum Python version.)3.11.*). (Pros: Prevents breaking changes when new versions of Python released + prevents environment drift. Cons: First build on any app using Poetry will likely fail, unless they are using a template that already has a stricter range set.)At the moment I'm leaning towards (4), since:
.python-versiontake priority overtool.poetry.dependencies.python, and so the error message can say to either adjust thetool.poetry.dependencies.pythonrange or create a.python-versionfile as an alternative (if they want to keep the wide range in the Poetry config).initcommand supports a--pythonoption, so we could always encourage users to usepoetry init --python '3.12.*'in our docs to save them having to fix the default version afterwards.poetry initwhich picks defaults more appropriate for an app vs a library (in addition to it setting a Python version like3.11.*instead of an unbounded range, it could also setpackage-mode = falsewhich would avoid all of the other boilerplate)We'll also want to factor in the
pyproject.tomlproject.requires-python field since that is what uv uses and so we may need to support that too in the future.See also:
.python-versionfile #6requires-pythonastral-sh/uv#7429GUS-W-9608268.
The text was updated successfully, but these errors were encountered: