Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support the Poetry package manager #7

Closed
edmorley opened this issue Feb 28, 2023 · 6 comments · Fixed by #261
Closed

Support the Poetry package manager #7

edmorley opened this issue Feb 28, 2023 · 6 comments · Fixed by #261
Assignees
Labels
enhancement New feature or request

Comments

@edmorley
Copy link
Member

edmorley commented Feb 28, 2023

We should support the Poetry package manager:
https://python-poetry.org/

See also:

@edmorley edmorley added the enhancement New feature or request label Feb 28, 2023
@larsakerson
Copy link

Hi @edmorley, are there any updates on the availability of this feature—or how to use it, if it is available?

@edmorley
Copy link
Member Author

edmorley commented Apr 3, 2023

@larsakerson Hi! This GitHub issue is still open, so the feature hasn't been added. Subscribe to this issue to get notified when it's added. (Which will be sometime later this year, but I don't have a timeline more specific than that.)

@edmorley
Copy link
Member Author

Quick status update - preparation for this has been happening in:

...and I have the actual Poetry support implementation itself working locally. It just needs some final tests and polish and then I should have a PR open for it at some point next week :-)

@edmorley edmorley self-assigned this Aug 30, 2024
edmorley added a commit that referenced this issue Sep 3, 2024
The Python package manager Poetry is now supported for installing app
dependencies:
https://python-poetry.org

To use Poetry apps must have a `poetry.lock` lockfile, which can be
created by running `poetry lock` locally, after adding Poetry config to
`pyproject.toml` (which can be done either manually or by using
`poetry init`). Apps must only have one package manager file (either
`requirements.txt` or `poetry.lock`, but not both) otherwise the
buildpack will abort the build with an error (which will help prevent
some of the types of support tickets we see in the classic buildpack).

Poetry is installed into a build-only layer, so is not available at
run-time to reduce image size. The app dependencies are installed into
a virtual environment (the same as for pip, after #257), which is on
`PATH` so does not need explicit activation when using the app image.
As such, use of `poetry run` or `poetry shell` is not required at
run-time to use dependencies in the environment.

When using Poetry, pip is not explicitly installed, since Poetry
includes its own bundled copy that it will use instead (for the small
number of Poetry operations for which it still calls out to pip, such
as package uninstalls).

Both the Poetry and app dependencies layers are cached, however, the
Poetry download/wheel cache is not cached, since using it is slower than
caching the dependencies layer (for more details see the comments on
`poetry_dependencies::install_dependencies`).

The `poetry install --sync` command is run using `--only main` so as to
only install the main dependencies group and not any other groups (such
as test/dev/... groups).

Relevant Poetry docs:
- https://python-poetry.org/docs/cli/#install
- https://python-poetry.org/docs/configuration/
- https://python-poetry.org/docs/managing-dependencies/#dependency-groups

Work that will be handled later:
- Support for selecting Python version via `tool.poetry.dependencies.python`:
  #260
- Build output and error messages polish/CX review (this will be
  performed when switching the buildpack to the new logging style).
- More detailed user-facing docs:
  #11

Closes #7.
GUS-W-9607867.
GUS-W-9608286.
GUS-W-9608295.
@edmorley
Copy link
Member Author

edmorley commented Sep 3, 2024

Poetry support PR now open: #261.

edmorley added a commit that referenced this issue Sep 4, 2024
The Python package manager Poetry is now supported for installing app
dependencies:
https://python-poetry.org

To use Poetry, apps must have a `poetry.lock` lockfile, which can be
created by running `poetry lock` locally, after adding Poetry config to
`pyproject.toml` (which can be done either manually or by using
`poetry init`). Apps must only have one package manager file (either
`requirements.txt` or `poetry.lock`, but not both) otherwise the
buildpack will abort the build with an error (which will help prevent
some of the types of support tickets we see in the classic buildpack
with users unknowingly mixing and matching pip + Pipenv).

Poetry is installed into a build-only layer (to reduce the final app
image size), so is not available at run-time. The app dependencies are
installed into a virtual environment (the same as for pip after #257,
for the reasons described in #253), which is on `PATH` so does not need
explicit activation when using the app image. As such, use of
`poetry run` or `poetry shell` is not required at run-time to use
dependencies in the environment.

When using Poetry, pip is not installed (possible thanks to #258), since
Poetry includes its own internal vendored copy that it will use instead
(for the small number of Poetry operations for which it still calls out
to pip, such as package uninstalls).

Both the Poetry and app dependencies layers are cached, however, the
Poetry download/wheel cache is not cached, since using it is slower than
caching the dependencies layer (for more details see the comments on
`poetry_dependencies::install_dependencies`).

The `poetry install --sync` command is run using `--only main` so as to
only install the main `[tool.poetry.dependencies]` dependencies group
from `pyproject.toml`, and not any of the app's other dependency groups
(such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`).

I've marked this `semver: major` since in the (probably unlikely) event
there are any early-adopter projects using this CNB that have both a
`requirements.txt` and `poetry.lock` then this change will cause them to
error (until one of the files is deleted).

Relevant Poetry docs:
- https://python-poetry.org/docs/cli/#install
- https://python-poetry.org/docs/configuration/
- https://python-poetry.org/docs/managing-dependencies/#dependency-groups

Work that will be handled later:
- Support for selecting Python version via `tool.poetry.dependencies.python`:
  #260
- Build output and error messages polish/CX review (this will be performed
  when switching the buildpack to the new logging style).
- More detailed user-facing docs:
  #11

Closes #7.
GUS-W-9607867.
GUS-W-9608286.
GUS-W-9608295.
@edmorley
Copy link
Member Author

edmorley commented Sep 4, 2024

Poetry support has now been released in v0.17.0 of this buildpack, and the Heroku CNB builder image updated with the new buildpack version, so Poetry builds now work with the latest version of the builder :-)

  1. git clone https://github.com/heroku/python-getting-started && cd python-getting-started
  2.  cat > pyproject.toml<<'EOF'
     [tool.poetry]
     package-mode = false
     
     [tool.poetry.dependencies]
     python = "3.12.*"
     django = "^5.1"
     gunicorn = "^23"
     dj-database-url = "^2"
     whitenoise = "^6"
     EOF
    
  3. poetry lock
  4. rm requirements.txt
  5. pack build --builder heroku/builder:24 poetry-test
$ pack build --builder heroku/builder:24 poetry-test
24: Pulling from heroku/builder
Digest: sha256:8a5252099ee2717fb69769909f1fca61ee9b4460faf564fab2c937a9f989253d
Status: Image is up to date for heroku/builder:24
24: Pulling from heroku/heroku
Digest: sha256:2363090f27c848b67a06a3a269dac13f042ffeed61af27432bd3bad360ed1d0e
Status: Image is up to date for heroku/heroku:24
===> ANALYZING
Image with name "poetry-test" not found
===> DETECTING
heroku/python   0.17.0
heroku/procfile 3.1.2
===> RESTORING
===> BUILDING

[Determining Python version]
Using Python version 3.12.5 specified in runtime.txt

[Installing Python]
Installing Python 3.12.5

[Installing Poetry]
Installing Poetry 1.8.3

[Installing dependencies using Poetry]
Creating virtual environment
Running 'poetry install --sync --only main'
Installing dependencies from lock file

Package operations: 8 installs, 0 updates, 0 removals

  - Installing asgiref (3.8.1)
  - Installing sqlparse (0.5.1)
  - Installing django (5.1.1)
  - Installing packaging (24.1)
  - Installing typing-extensions (4.12.2)
  - Installing dj-database-url (2.2.0)
  - Installing gunicorn (23.0.0)
  - Installing whitenoise (6.7.0)

[Generating Django static files]
Running 'manage.py collectstatic'

1 static file symlinked to '/workspace/staticfiles', 1 post-processed.

[Discovering process types]
Procfile declares types -> web
===> EXPORTING
Adding layer 'heroku/python:python'
Adding layer 'heroku/python:venv'
Adding layer 'buildpacksio/lifecycle:launch.sbom'
Adding 1/1 app layer(s)
Adding layer 'buildpacksio/lifecycle:launcher'
Adding layer 'buildpacksio/lifecycle:config'
Adding layer 'buildpacksio/lifecycle:process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving poetry-test...
*** Images (a4be12101625):
      poetry-test
Adding cache layer 'heroku/python:poetry'
Adding cache layer 'heroku/python:python'
Adding cache layer 'heroku/python:venv'
Successfully built image poetry-test

@edmorley
Copy link
Member Author

edmorley commented Sep 4, 2024

Oh and support for selecting the Python version using tool.poetry.dependencies.python is being tracked in:
#260

In the meantime, use runtime.txt to control the Python version (or omit runtime.txt to use the default Python version).

Note that once support for tool.poetry.dependencies.python lands, it's likely that some builds will error due to use of "unsafe" version ranges in tool.poetry.dependencies.python (see the linked issue for more details). As such, if you are switching to Poetry now and want to prevent potential build errors in the future, I'd recommend using a tool.poetry.dependencies.python value like 3.12.* instead of ^3.12.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants