Speed up update-python3-dependencies
#7234
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Status
Ready for review
Description of Changes
The
make update-python3-dependencies
step is slow for two main reasons: container image layering and pip-compile.Currently, each
pip-compile
step is run in a separate dev-shell, which means after each one, if anything changed in the requirements/ folder, it needs to rebuild the slim image and reinstall all the pip dependencies.Now I've moved the steps to a separate script that execute in a single dev-shell, so no image rebuilding happens during the updates, it'll just be needed once afterwards.
For an additional boost, switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the
--allow-unsafe
because uv is entirely independent of setuptools and pip-tools.uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed.
Overall
make update-python3-dependencies
now takes seconds to run instead of minutes \o/Refs freedomofpress/securedrop-tooling#16.
Testing
How should the reviewer test this PR?
make update-python3-dependencies
yourself.Deployment
Any special considerations for deployment? n/a
Checklist
make lint
) and tests (make test
) pass in the development containermake -C admin test
) pass in the admin development container