Skip to content

Commit

Permalink
[pip] Remove make targets for pinned requirements files (#14604)
Browse files Browse the repository at this point in the history 
These make targets have been more trouble than they are worth. Since
both the requirements files and the pinned requirements files that they
generate are checked into the repo, updates to these files through PRs
don't play well with how make tracks timestamps, since my local git may
update them in different orders. In general, I've never had a need to
only update a single requirements file and since these are technically
dependencies of say, the Batch image, building the batch image can
incidentally trigger a re-generation of these files when I never
actually changed the dependencies. This PR removes those targets and
does a normal refresh of the requirements, as well as updates the
documentation on how we handle pip dependencies.
  • Loading branch information
@daniel-goldstein
daniel-goldstein authored Jul 9, 2024
1 parent fc14b56 commit ab5c1ff
Show file tree
Hide file tree
Showing 10 changed files with 680 additions and 213 deletions.
24 changes: 2 additions & 22 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,36 +99,16 @@ install-dev-requirements:
-r batch/pinned-requirements.txt \
-r ci/pinned-requirements.txt

hail/python/hailtop/pinned-requirements.txt: hail/python/hailtop/requirements.txt
.PHONY: generate-pip-lockfiles
generate-pip-lockfiles:
./generate-pip-lockfile.sh hail/python/hailtop

hail/python/pinned-requirements.txt: hail/python/hailtop/pinned-requirements.txt hail/python/requirements.txt
./generate-pip-lockfile.sh hail/python

hail/python/dev/pinned-requirements.txt: hail/python/pinned-requirements.txt hail/python/dev/requirements.txt
./generate-pip-lockfile.sh hail/python/dev

gear/pinned-requirements.txt: hail/python/pinned-requirements.txt hail/python/dev/pinned-requirements.txt hail/python/hailtop/pinned-requirements.txt gear/requirements.txt
./generate-pip-lockfile.sh gear

web_common/pinned-requirements.txt: gear/pinned-requirements.txt web_common/requirements.txt
./generate-pip-lockfile.sh web_common

batch/pinned-requirements.txt: web_common/pinned-requirements.txt batch/requirements.txt
./generate-pip-lockfile.sh batch

ci/pinned-requirements.txt: web_common/pinned-requirements.txt ci/requirements.txt
./generate-pip-lockfile.sh ci

.PHONY: generate-pip-lockfiles
generate-pip-lockfiles: hail/python/hailtop/pinned-requirements.txt
generate-pip-lockfiles: hail/python/pinned-requirements.txt
generate-pip-lockfiles: hail/python/dev/pinned-requirements.txt
generate-pip-lockfiles: gear/pinned-requirements.txt
generate-pip-lockfiles: web_common/pinned-requirements.txt
generate-pip-lockfiles: batch/pinned-requirements.txt
generate-pip-lockfiles: ci/pinned-requirements.txt

$(HAILTOP_VERSION):
$(MAKE) -C hail python/hailtop/hail_version

Expand Down
102 changes: 84 additions & 18 deletions batch/pinned-requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,43 +1,109 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --python-version 3.9 --python-platform linux batch/requirements.txt --output-file=batch/pinned-requirements.txt
aiodocker==0.21.0
aiodocker==0.22.0
# via -r batch/requirements.txt
aiohttp==3.9.5
# via aiodocker
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# aiodocker
aiorwlock==1.4.0
# via -r batch/requirements.txt
aiosignal==1.3.1
# via aiohttp
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# aiohttp
async-timeout==4.0.3
# via aiohttp
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# -r batch/requirements.txt
# aiohttp
attrs==23.2.0
# via aiohttp
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# aiohttp
dictdiffer==0.9.0
# via -r batch/requirements.txt
frozenlist==1.4.1
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# aiohttp
# aiosignal
idna==3.7
# via yarl
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# yarl
multidict==6.0.5
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# aiohttp
# yarl
numpy==1.26.4
# via pandas
packaging==24.0
# via plotly
# via
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# pandas
packaging==24.1
# via
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# plotly
pandas==2.2.2
# via
# -c batch/../hail/python/pinned-requirements.txt
# -r batch/requirements.txt
plotly==5.22.0
# via
# -c batch/../hail/python/pinned-requirements.txt
# -r batch/requirements.txt
python-dateutil==2.9.0.post0
# via pandas
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# pandas
pytz==2024.1
# via pandas
# via
# -c batch/../hail/python/pinned-requirements.txt
# pandas
six==1.16.0
# via python-dateutil
tenacity==8.3.0
# via plotly
typing-extensions==4.11.0
# via aiodocker
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# python-dateutil
tenacity==8.4.2
# via
# -c batch/../hail/python/pinned-requirements.txt
# plotly
tzdata==2024.1
# via pandas
# via
# -c batch/../hail/python/pinned-requirements.txt
# pandas
yarl==1.9.4
# via aiohttp
# via
# -c batch/../gear/pinned-requirements.txt
# -c batch/../hail/python/dev/pinned-requirements.txt
# -c batch/../hail/python/pinned-requirements.txt
# -c batch/../web_common/pinned-requirements.txt
# aiohttp
75 changes: 58 additions & 17 deletions ci/pinned-requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,71 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --python-version 3.9 --python-platform linux ci/requirements.txt --output-file=ci/pinned-requirements.txt
certifi==2024.2.2
# via requests
certifi==2024.6.2
# via
# -c ci/../gear/pinned-requirements.txt
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# requests
cffi==1.16.0
# via cryptography
# via
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# cryptography
charset-normalizer==3.3.2
# via requests
# via
# -c ci/../gear/pinned-requirements.txt
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# requests
click==8.1.7
# via zulip
cryptography==42.0.7
# via pyjwt
# via
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# zulip
cryptography==42.0.8
# via
# -c ci/../hail/python/pinned-requirements.txt
# pyjwt
distro==1.9.0
# via zulip
gidgethub==5.3.0
# via -r ci/requirements.txt
idna==3.7
# via requests
# via
# -c ci/../gear/pinned-requirements.txt
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# -c ci/../web_common/pinned-requirements.txt
# requests
pycparser==2.22
# via cffi
# via
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# cffi
pyjwt==2.8.0
# via gidgethub
requests==2.32.2
# via zulip
typing-extensions==4.11.0
# via zulip
# via
# -c ci/../hail/python/pinned-requirements.txt
# gidgethub
requests==2.32.3
# via
# -c ci/../gear/pinned-requirements.txt
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# zulip
typing-extensions==4.12.2
# via
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# zulip
uritemplate==4.1.1
# via gidgethub
urllib3==1.26.18
# via requests
# via
# -c ci/../gear/pinned-requirements.txt
# gidgethub
urllib3==1.26.19
# via
# -c ci/../gear/pinned-requirements.txt
# -c ci/../hail/python/dev/pinned-requirements.txt
# -c ci/../hail/python/pinned-requirements.txt
# requests
zulip==0.9.0
# via -r ci/requirements.txt
13 changes: 6 additions & 7 deletions dev-docs/pip-dependencies.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,11 @@ This allows deterministic builds in our Dockerfiles. When adding a dependency
to a `requirements.txt` file, run

```bash
make generate-pip-lockfile
make generate-pip-lockfiles
```

to regenerate the `pinnned-requirements.txt` file to one compatible with the
new requirements. Note that the full dependency tree for a pip package can
differ on different operating systems. At the time of writing, the dependencies
for the hail pip package should be identical on MacOS and Linux, but the services
dependencies differ so are generated in a linux docker container as that's the
platform on which the services modules will run.
to regenerate the `pinnned-requirements.txt` files in the repository.
Note that the full dependency tree for a pip package can
differ on different operating systems. All services and docker images in CI
use the fully pinned requirements on a Linux platform, so the pinned requirements
files are only generated for Linux.
Loading

0 comments on commit ab5c1ff

Please sign in to comment.