Inconsistent default behavior of include

[radoering]

Before diving into the issue, a short clarification:

1. There are packages includes, which default to sdist and wheel.
2. There are includes, which (according to the docs) default to only sdist.

These different defaults can be confusing, but might make sense because you normally want to include packages in sdists and wheels but additional data like tests and a changelog only in the sdist.

Issue

The behavior for includes (the second one) is different for files and directories:

• files are included in both sdist and wheel
• directories are only included in sdists

Considering that the docs were "fixed" recently in #8852 to describe the behavior for directories - previously, it described the behavior for files - I wondered which behavior is correct and which is the bug?

Events of confusion in chronologically order:

• Add support of inline tables in include on pyproject.toml. poetry-core#6 added support for includes and describes that they should be only sdist per default. This PR has already contained the bug.
• Includes default format poetry-core#114 changed the default behavior for packages includes (first one), but confusingly added a test for includes (second one) - actually testing the bug instead of the change. Thus, we cannot trust the test.
• Add documentation on inline tables in include. Resolves: Core #8. #2268 added docs that it defaults to sdist and wheel.
• Fix documentation for default format option for include and exclude #8852 changed the docs that it defaults to only sdist.

We can be sure that the initial intention was to default to "only sdist". However, I am not sure if the behavior should be changed to "sdist and wheel" later or if it was just a confusion between the two types of includes.

Thus, the big question: What should the default be?

• sdist and wheel for package includes (first type) and sdist only for includes (second type)
  • pro: You normally want to include packages in sdists and wheels but additional data like tests and a changelog only in the sdist
  • contra: different defaults
• sdist and wheel for both types of includes
  • pro: same defaults
  • contra: maybe improper default for one type

[radoering]

After taking a closer look at it, here are a few more arguments:

default to "only sdist"

• When installing a wheel with includes, the includes are unpacked straight into site-packages, which does not makes sense for a top level changelog or tests. (thanks @Secrus)

default to "sdist and wheel"

• You need include to include vcs excluded files and it is weird to include such files in an sdist but not in a wheel:
  • vcs excluded (e.g. generated) files are excluded automatically from package include, you have to overwrite the exclude behavior via include
  • docs:

    poetry/docs/pyproject.md

    Line 632 in b61e6bd

    If a VCS is being used for a package, the exclude field will be seeded with the VCS’ ignore settings (`.gitignore` for git for example).

  • pyproject.toml from tests: https://github.com/python-poetry/poetry-core/blob/b08f326bc1348fc7d8cde25b41fff35e57257745/tests/masonry/builders/fixtures/with-include/pyproject.toml#L36
• It is really weird if include and exclude would have different defaults:
  • pyproject.toml from tests: https://github.com/python-poetry/poetry-core/blob/b08f326bc1348fc7d8cde25b41fff35e57257745/tests/masonry/builders/fixtures/exclude_nested_data_toml/pyproject.toml#L12-L13

So it looks like there are good reasons for either direction...

In general, I think it is easier to understand that all sorts of include (and exlcude) default to all formats (i.e. sdist and wheel). However, I also see the danger that this results in changelogs and tests being unintentionally included in wheels.

[radoering]

@johnthagen Since you often give good advice on usability and this is a difficult choice, I'd appreciate your opinion. (In case, you have never used this feature and have no opinion that is also fine, of course.)

[johnthagen]

@radoering Thanks for the ping. I primarily use Poetry for applications rather than for packaging sdists/wheels for libraries. Because of this, I have not used these features of Poetry before so my opinion is not very relevant.

For example, I was a big fan of

• Introduce non-package-mode #8650

Because this is actually how I use Poetry these days.