Switch to using poetry (#374)

* Switch to using poetry

* Add changelog entry

* Move changefile to actual PR number

* Try with poetry in workflow

* Add tox to pyproject

* Update poetry.lock

* Update workflow to use specific python versions

* Update poetry.lock

* Use path when running tests in CI

* Try including tests

* Try old way of running tests

* Use old way for old deps tests as well

* Update dockerfile to use poetry

* Update poetry to a much newer version

.github/workflows/pipeline.yml

@@ -8,22 +8,28 @@ jobs:
   check-code-style:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v4
+      - uses: actions/checkout@v4
+
+      - name: Setup Poetry
+        uses: matrix-org/setup-python-poetry@v1
         with:
+          install-project: "false"
           python-version: "3.11"
-      - run: python -m pip install tox
-      - run: tox -e check_codestyle
+
+      - run: poetry run tox -e check_codestyle
 
   check-types-mypy:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v4
+      - uses: actions/checkout@v4
+
+      - name: Setup Poetry
+        uses: matrix-org/setup-python-poetry@v1
         with:
+          install-project: "false"
           python-version: "3.11"
-      - run: python -m pip install tox
-      - run: tox -e check_types
+
+      - run: poetry run tox -e check_types
 
   run-unit-tests:
     name: Unit tests

CONTRIBUTING.md

@@ -4,35 +4,33 @@ Everyone is welcome to contribute code to Sygnal, provided you are willing to
 license your contributions under the same license as the project itself. In
 this case, the [Apache Software License v2](LICENSE).
 
-### Create a virtualenv
+### Installing dependencies
 
 To contribute to Sygnal, ensure you have Python 3.8 or newer and then run:
 
-```bash
-python3 -m venv venv
-./venv/bin/pip install -e '.[dev]'
-```
+Sygnal uses the [poetry](https://python-poetry.org/) project to manage its dependencies
+and development environment. Once you have installed Python 3 and added the
+source, you should install `poetry`.
+Of their installation methods, we recommend
+[installing `poetry` using `pipx`](https://python-poetry.org/docs/#installing-with-pipx),
 
-This creates an isolated virtual Python environment ("virtualenv") just for
-use with Sygnal, then installs Sygnal along with its dependencies, and lastly
-installs a handful of useful tools
+```shell
+pip install --user pipx
+pipx install poetry
+```
 
-If you get `ConnectTimeoutError`, this is caused by slow internet whereby
-`pip` has a default time out of _15 sec_. You can specify a larger timeout
-by passing `--timeout 120` to the `pip install` command above.
+but see poetry's [installation instructions](https://python-poetry.org/docs/#installation)
+for other installation methods.
 
-Finally, activate the virtualenv by running:
+Next, open a terminal and install dependencies as follows:
 
-```bash
-source ./venv/bin/activate
+```sh
+cd path/where/you/have/cloned/the/repository
+poetry install
 ```
 
-Be sure to do this _every time_ you open a new terminal window for working on
-Sygnal. Activating the venv ensures that any Python commands you run (`pip`,
-`python`, etc.) use the versions inside your venv, and not your system Python.
-
-When you're done, you can close your terminal or run `deactivate` to disable
-the virtualenv.
+This will install the runtime and developer dependencies for the project.  Be sure to check
+that the `poetry install` step completed cleanly.
 
 ### Run the tests
 
@@ -56,6 +54,25 @@ ___________________________________ summary ___________________________________
 
 Then all is well and you're ready to work!
 
+You can also directly run the tests using poetry.
+
+```sh
+poetry run trial tests
+```
+
+You can run unit tests in parallel by specifying `-jX` argument to `trial` where `X` is the number of parallel runners you want. To use 4 cpu cores, you would run them like:
+
+```sh
+poetry run trial -j4 tests
+```
+
+If you wish to only run *some* unit tests, you may specify
+another module instead of `tests` - or a test class or a method:
+
+```sh
+poetry run trial tests.test_apns.ApnsTestCase.test_expected
+```
+
 ## How to contribute
 
 The preferred and easiest way to contribute changes is to fork the relevant
@@ -90,13 +107,9 @@ Many of the conventions are enforced by scripts which are run as part of the
 [continuous integration system](#continuous-integration-and-testing).
 
 To help check and fix adherence to the code style, you can run `tox`
-locally. You'll need Python 3.8 or later, and a virtual environment configured and
-active:
+locally. You'll need Python 3.8 or later:
 
 ```bash
-# Activate the virtual environment
-source ./venv/bin/activate
-
 # Run the code style check
 tox -e check_codestyle
 
@@ -114,6 +127,10 @@ Please ensure your changes match the cosmetic style of the existing project,
 and **never** mix cosmetic and functional changes in the same commit, as it
 makes it horribly hard to review otherwise.
 
+## Further information on poetry
+
+See the information provided in the [Synapse docs](https://github.com/element-hq/synapse/blob/master/docs/development/dependencies.md).
+
 ## Changelog
 
 All changes, even minor ones, need a corresponding changelog / newsfragment

RELEASING.md

@@ -1,35 +1,36 @@
 0. Consider whether this release will affect any customers, including those on
 EMS, and warn them beforehand - in case they need to upgrade quickly.
 
-1. Set a variable to the version number for convenience:
+1. Update the version number in pyproject.toml.
+2. Set a variable to the version number for convenience:
    ```sh
    ver=x.y.z
    ```
-1. Update the changelog:
+3. Update the changelog:
    ```sh
    towncrier --version=$ver
    ```
-1. Push your changes:
+4. Push your changes:
    ```sh
    git add -u && git commit -m $ver && git push
    ```
-1. Sanity-check the
+5. Sanity-check the
    [changelog](https://github.com/matrix-org/sygnal/blob/master/CHANGELOG.md)
    and update if need be.
-1. Create a signed tag for the relese:
+6. Create a signed tag for the relese:
    ```sh
    git tag -s v$ver
    ```
    Base the tag message on the changelog.
-1. Push the tag:
+7. Push the tag:
    ```sh
    git push origin tag v$ver
    ```
    Pushing a tag on GitHub will automatically trigger a build in Docker Hub and
    the resulting image will be published using the same tag as git.
-1. Create release on GH project page:
+8. Create release on GH project page:
    ```sh
    xdg-open https://github.com/matrix-org/sygnal/releases/edit/v$ver
    ```
-1. Notify #sygnal:matrix.org, #synapse-dev:matrix.org and EMS that a new
+9. Notify #sygnal:matrix.org, #synapse-dev:matrix.org and EMS that a new
    release has been published.

changelog.d/374.misc

@@ -0,0 +1 @@
+Switch over to use poetry & add lock file to version control.

docker/Dockerfile

@@ -6,26 +6,81 @@
 #    docker build -f docker/Dockerfile .
 #
 
+ARG PYTHON_VERSION=3.11
+
+###
+### Stage 0: generate requirements.txt
+###
+# We hardcode the use of Debian bookworm here because this could change upstream.
+FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm as requirements
+
+# We install poetry in its own build stage to avoid its dependencies conflicting with
+# sygnal's dependencies.
+RUN --mount=type=cache,target=/root/.cache/pip \
+  pip install --user "poetry==1.8.3"
+
+WORKDIR /sygnal
+
+# Copy just what we need to run `poetry export`...
+COPY pyproject.toml poetry.lock /sygnal/
+
+# If specified, we won't verify the hashes of dependencies.
+# This is only needed if the hashes of dependencies cannot be checked for some
+# reason, such as when a git repository is used directly as a dependency.
+ARG TEST_ONLY_SKIP_DEP_HASH_VERIFICATION
+
+# If specified, we won't use the Poetry lockfile.
+# Instead, we'll just install what a regular `pip install` would from PyPI.
+ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE
+
+# Export the dependencies, but only if we're actually going to use the Poetry lockfile.
+# Otherwise, just create an empty requirements file so that the Dockerfile can
+# proceed.
+RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
+  /root/.local/bin/poetry export -o /sygnal/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}; \
+  else \
+  touch /sygnal/requirements.txt; \
+  fi
+
 ###
-### Stage 0: builder
+### Stage 1: builder
 ###
-FROM python:3.11-slim as builder
+FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm as builder
 
-# Install git; Sygnal uses it to obtain the package version from the state of the
-# git repository.
-RUN apt-get update && apt-get install -y git
+# To speed up rebuilds, install all of the dependencies before we copy over
+# the whole sygnal project, so that this layer in the Docker cache can be
+# used while you develop on the source.
+#
+# This is aiming at installing the `[tool.poetry.depdendencies]` from pyproject.toml.
+COPY --from=requirements /sygnal/requirements.txt /sygnal/
+RUN --mount=type=cache,target=/root/.cache/pip \
+  pip install --prefix="/install" --no-deps --no-warn-script-location -r /sygnal/requirements.txt
 
-# install sygnal and all of the python deps to /install.
+# Copy over the rest of the sygnal source code.
+COPY sygnal /sygnal/sygnal/
+# ... and what we need to `pip install`.
+COPY pyproject.toml README.md /sygnal/
 
-COPY . /sygnal/
+# Repeat of earlier build argument declaration, as this is a new build stage.
+ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE
 
-RUN pip install --prefix="/install" --no-warn-script-location /sygnal
+# Install the sygnal package itself.
+# If we have populated requirements.txt, we don't install any dependencies
+# as we should already have those from the previous `pip install` step.
+RUN --mount=type=cache,target=/sygnal/target,sharing=locked \
+  --mount=type=cache,target=${CARGO_HOME}/registry,sharing=locked \
+  if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
+  pip install --prefix="/install" --no-deps --no-warn-script-location /sygnal; \
+  else \
+  pip install --prefix="/install" --no-warn-script-location /sygnal; \
+  fi
 
 ###
-### Stage 1: runtime
+### Stage 2: runtime
 ###
 
-FROM python:3.11-slim
+FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm
+
 COPY --from=builder /install /usr/local
 
 EXPOSE 5000/tcp