Add support for the project section (PEP 621)

[radoering]

Pull Request Check List

Resolves: #3332
Resolves: python-poetry/roadmap#3
Requires: python-poetry/poetry-core#708

should not be squashed

• Added tests for changed code.
• Updated documentation for changed code.

Current state:

• Full compatibility with Add support for the project section (PEP 621) poetry-core#708
• All the commands should work, especially check, lock, install, build, add, remove, init, new
• Updated documentation:
  • See preview
  • Explanation of project fields and use cases for dynamic (version, readme, classifiers) in the section "The pyproject.toml file".
  • Explanation of the difference between project.dependencies and tool.poetry.dependencies in the section "Dependency specification".
  • Updated examples to use project fields instead of tool.poetry fields.

You can install this branch with pipx as follows:

pipx install --suffix _pep621 git+https://github.com/radoering/poetry.git@pep621-support

This will install Poetry with the suffix "_pep621" so that you have to call poetry_pep621 instead of poetry. (The suffix is arbitrary, you can choose a different one.)

There is no automatic conversion of existing pyproject.toml files (yet?) but poetry check will give you information which fields should be transferred from tool.poetry to project.

[github-actions]

Deploy preview for website ready!

✅ Preview
https://website-870gzwxbb-python-poetry.vercel.app

Built with commit 28bbbf3.
This pull request is being automatically deployed with vercel-action

[johnthagen]

After this is released, Poetry can be added to the PyPA packaging tutorial. 🚀

• https://packaging.python.org/en/latest/tutorials/packaging-projects/

[Evan-Zhao]

First of all, thank you for the great work!

I'm a bit confused by the entire extra business (which is pretty complicated itself), and especially how poetry add interacts with extras. Disclaimer: I may have wrong understanding of extras or optionals.

For this example pyproject.toml, say now I would like to add an extra feature called "dev" with dependencies black and Cython. This way pip install tvm[dev] will also install black and Cython, which a tvm developer will need (but not general users). In other words I want to do poetry add ??? black Cython, and Poetry should add such a line:

[project.optional-dependencies]
# ...
dev = ["cython (>=3.0.10,<4.0.0)", "black >= 24"]   # added

So I go to poetry add --help and find two relevant options, but it turns out neither does what I want here.
One is --optional, which specifies the dependencies are optional using a tool.poetry.dependencies specification. Makes perfect sense. The other one, --extras <extra>, is harder to follow.

Suppose I already work with tvm[highlight] (that is, with the extra highlight on) in my own environment.
If I do poetry add <dep> without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps. It wasn't clear to me from the help, but this behavior is logical. But now I do poetry add -E highlight Cython, and Poetry adds this:

dependencies = [
  # ...
  "cython[highlight] (>=3.0.10,<4.0.0)"]  # added

where highlight becomes cython's extra. Is that expected? I thought if I want to add a dep with an extra I'd do poetry add "dep[extra]" instead.

It looks to me we need a command that "given existing extras e_1, e_2, ... e_n, add deps dep_1, dep_2, ..., dep_n into an optional extra e_new (if not given then add to global dependencies)",
so something like poetry add [--current-extras e]+ [--target-extra e]? [dep]+ (not actually proposing this syntax).

• In developing my project I keep multiple current-extras on, and it's kind of clumsy to always enter all of them for each add command -- is it possible at all to auto-deduce that from the lock file?
• Also I can't speak for how all of this interact with Poetry optionals and dep groups, since I'm not using these yet, but I'd love to help discuss should complexity arises there.

[radoering]

If I do poetry add without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps.

That's unfortunate but not easy to change as we do not keep track of which extras (of your project) were installed. Furhter, it is not specific to this PR and thereby out of scope here. I suppose there is already an issue about this but I can't find it right now. Feel free to create a new issue. (In case it's a duplicate someone will tell you.)

If I do poetry add without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps.

This conclusion is not correct. poetry add does not consider extras of your project for installing. You can probably work around this by using poetry add --lock ... to only lock (and not install in your environment) and in the end run poetry install -E highlight to install everything including your project's extra.

where highlight becomes cython's extra. Is that expected?

That's expected. See poetry add --help or the docs: "Extras to activate for the dependency."

I thought if I want to add a dep with an extra I'd do poetry add "dep[extra]" instead.

Both are equivalent.

It looks to me we need a command

--optional just adds an optional dependency but does not add it to an extra. Since that does not make sense anymore in the project section, I changed --optional to require an extra name to which the dependency is added, so now you can do: poetry add --optional highlight Cython to add Cython to the extra highlight.

In developing my project I keep multiple current-extras on, and it's kind of clumsy to always enter all of them for each add command -- is it possible at all to auto-deduce that from the lock file?

In the lockfile, only extras of dependencies are fix. All extras of your project are locked and you can/must decide which to install when running poetry install.

Also I can't speak for how all of this interact with Poetry optionals and dep groups, since I'm not using these yet

which a tvm developer will need (but not general users).

Maybe, you should consider using dependency groups instead of extras. However, dependency groups are only available during development when using Poetry and not visible in the sdist/wheel or to other tools.

All in all, there is some room for improvement considering extras but everything that goes beyond my recent change regarding --optional should probably be discussed in a separate issue.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.