Add Support for Dynamic Dependencies in pyproject.toml

[zz1874]

Closes #347

Description
This PR introduces support for dynamic dependencies declared in the pyproject.toml file under tool.setuptools.dynamic section.

In the past, our project had a feature where it did not recurse into subdirectories under the project root (.) to detect requirements files. This PR enables our project to detect requirements files located in any directory as long as they are declared under tool.setuptools.dynamic section.

Changes

• Modified the existing code structure to integrate dynamic dependencies into the dependency parsing flow.
• Implemented handling for both mandatory and optional dynamic dependencies.
• Checked the existence of dynamic dependencies' requirements files.

Usage
In your project's pyproject.toml file, define dynamic dependencies under the tool.setuptools.dynamic section via setuptools or setuptools-scm. For example:

[tool.setuptools.dynamic]
dependencies = { file = [".config/requirements-dynamic.txt"] }
optional-dependencies.docs = { file = [".config/optional-docs-reqs.txt"] }

Run fawltydeps under your project as usual. The PR introduces support for dynamic dependencies, and they will be automatically detected and processed.

Testing
Adding unit tests is still WIP.

[jherland]

(Not a full review, but I had some free minutes, and wanted to take a quick look...)

I love how this is coming together, and how little code you have to change to make this work. 😃

One question:
Should we check/require the project.dynamic directive before looking at the tool.setuptools.dynamic? I'm looking at https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html to understand how this should work, and it seems to me that tool.setuptools.dynamic should not be looked at when the corrsponding key in project.dynamic is missing?

[zz1874]

@jherland Thanks for the review!

Should we check/require the project.dynamic directive before looking at the tool.setuptools.dynamic?

Yes I've also spotted this issue. What I was thinking is that it takes the "same" effort to look at either project.dynamic or tool.setuptools.dynamic. But of course we can look at project.dynamic first as an initial condition.

[Nour-Mws]

Looks good so far! I think the tests will help us see better if anything is missing.
As I said offline, being able to add these requirements files to the traversal object would have been a neat solution but we unfortunately don't have a direct way to append to the list of dependencies files to traverse from this function.

[jherland]

@zz1874:

Should we check/require the project.dynamic directive before looking at the tool.setuptools.dynamic?

Yes I've also spotted this issue. What I was thinking is that it takes the "same" effort to look at either project.dynamic or tool.setuptools.dynamic. But of course we can look at project.dynamic first as an initial condition.

I believe we should look at project.dynamic as an initial condition, yes. According to https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata (which is the closest thing I have found to an authoritative spec here), the tool.setuptools.dynamic only correspond to what is found/requested in project.dynamic. If project.dynamic is missing I think we should NOT look for corresponding entries in tool.setuptools.dynamic at all.

@Nour-Mws:

As I said offline, being able to add these requirements files to the traversal object would have been a neat solution but we unfortunately don't have a direct way to append to the list of dependencies files to traverse from this function.

I agree that we currently do not have a good way to feed this information back into the traversal. But I don't know if it would be very useful to update the traversal, since the traversal phase is already done/finished before we parse this pyproject.toml and discover these dynamic fields?

Currently, since we don't parse files during the traversal phase, we would not expect these extra requirements files to show up in the --list-sources output (unless they coincidentally are also found by the traversal).

If we really want them to show up in --list-sources, we would have to move (some of) our pyproject.toml parsing into the traversal stage, and I'm not sure it's ultimately worth it?

[Nour-Mws]

Currently, since we don't parse files during the traversal phase, we would not expect these extra requirements files to show up in the --list-sources output (unless they coincidentally are also found by the traversal).

If we really want them to show up in --list-sources, we would have to move (some of) our pyproject.toml parsing into the traversal stage, and I'm not sure it's ultimately worth it?

@jherland, I don't think it's worth the hassle really (i.e. parsing pyproject.toml in the traversal stage). On principle, yes, I think these requirements files should appear in the sources. You're getting the requirements in these files by parsing them, and not directly from pyproject.toml so I would argue they are a direct source. But I don't think this is worth spending any time on.

[Nour-Mws]

I'd just like to say I really admire the way you've been able to get up and running on the test suite!
Great work!
I've left a couple of comments. Some are nitpicky (and I flagged them as such) but others would need more consideration.

[jherland]

As @Nour-Mws already said: Great work, and I'm also impressed by how quickly you're getting up to speed with everything here. 😄

There's still some way to go before all the corner cases have been covered, I think, and adding more test cases is IMHO the key to getting there. Therefore I'm not too worried about spending some time/discussion to get the basic test infrastructure right. It is clear that you have uncovered some missing pieces in the fake_project fixture, and I think getting those parts right will make it much easier to add more tests.

[Nour-Mws]

With PR #354 merged, the CI should now pass once this is rebased on main.

[jherland]

There's one discussion where I just posted a comment, and left unresolved. Except for that, I think all other comments have been resolved, and I'm very happy to see this merged. 😄 Great work!

[Nour-Mws]

Great improvements! Thanks!
I've added one comment here to second @jherland's suggestion of adding two more tests. Once added, the two remaining conversations can be closed and this merged!

[Nour-Mws]

This is ready to be merged!