Merge tag 'v0.9.3'

Sygnal 0.9.3 (2021-04-22)
=========================

Features
--------

- Prevent the push key from being rejected for temporary errors and oversized payloads, add TTL logging, and support `events_only` push data flag. ([\matrix-org#212](matrix-org#212))
- WebPush: add support for Urgency and Topic header ([\matrix-org#213](matrix-org#213))

Bugfixes
--------

- Fix a long-standing bug where invalid JSON would be accepted over the HTTP interfaces. ([\matrix-org#216](matrix-org#216))
- Limit the size of requests received from HTTP clients. ([\matrix-org#220](matrix-org#220))

Updates to the Docker image
---------------------------

- Remove manually added GeoTrust Root CA certificate from docker image as Apple is no longer using it. ([\matrix-org#208](matrix-org#208))

Improved Documentation
----------------------

- Make `CONTIBUTING.md` more explicit about how to get tests passing. ([\matrix-org#188](matrix-org#188))
- Update `CONTRIBUTING.md` to specify how to run code style and type checks with Tox, and add formatting to code block samples. ([\matrix-org#193](matrix-org#193))
- Document how to work around pip installation timeout errors. Contributed by Omar Mohamed. ([\matrix-org#215](matrix-org#215))

Internal Changes
----------------

- Update Tox to run in the installed version of Python (instead of specifying Python 3.7) and to consider specific paths and folders while running checks, instead of the whole repository (potentially including unwanted files and folders, e.g. the virtual environment). ([\matrix-org#193](matrix-org#193))
- Make development dependencies available as extras. Contributed by Hillery Shay. ([\matrix-org#194](matrix-org#194))
- Update `setup.py` to specify that a minimum version of Python greater or equal to 3.7 is required. Contributed by Tawanda Moyo. ([\matrix-org#207](matrix-org#207))
- Port CI checks to Github Actions. ([\matrix-org#210](matrix-org#210), [\matrix-org#219](matrix-org#219))
- Upgrade development dependencies. Contributed by Omar Mohamed ([\matrix-org#214](matrix-org#214))
- Set up `coverage.py` to run in tox environment, and add html reports ([\matrix-org#217](matrix-org#217))

Change-Id: I14ae821ff2a1562e91fd87ce25f73baaa0b9430b

.github/workflows/changelog_check.yml

@@ -0,0 +1,16 @@
+name: Changelog
+on: [pull_request]
+
+jobs:
+  check-newsfile:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+          ref: ${{github.event.pull_request.head.sha}}
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.7"
+      - run: python -m pip install towncrier
+      - run: python -m towncrier.check

.github/workflows/pipeline.yml

@@ -0,0 +1,64 @@
+name: Linting and Tests
+on: [push]
+jobs: 
+  check-code-style:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.7"
+      - run: python -m pip install tox
+      - run: tox -e check_codestyle
+
+  check-types-mypy:
+    runs-on: ubuntu-latest
+    steps: 
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.7"
+      - run: python -m pip install tox  
+      - run: tox -e check_types
+
+  run-unit-tests-sqlite:
+    needs: [check-code-style, check-types-mypy]
+    runs-on: ubuntu-latest
+    steps: 
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.7"
+      - run: python -m pip install -e .
+      - run: trial tests
+
+  run-unit-tests-postgres:
+    needs: [check-code-style, check-types-mypy]
+    runs-on: ubuntu-latest
+
+    services:
+      postgres:
+        image: postgres:11
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_PASSWORD: "postgres"
+          POSTGRES_INITDB_ARGS: "--lc-collate C --lc-ctype C --encoding UTF8"
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5  
+  
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.7"
+      - run: python -m pip install -e .
+      - run: trial tests
+        env:
+          TEST_USE_POSTGRES: "yes"
+          TEST_POSTGRES_USER: "postgres"
+          TEST_POSTGRES_PASSWORD: "postgres"
+          TEST_POSTGRES_HOST: "localhost"

.gitignore

@@ -4,10 +4,15 @@ var/
 sygnal.pid
 sygnal.db
 _trial_temp*
+.coverage*
 
+/venv/
+/.venv/
 /.idea
 /.eggs
 /*.egg-info
 /build
 /dist
 /.tox
+/.python-version
+/htmlcov

CHANGELOG.md

@@ -1,3 +1,104 @@
+Sygnal 0.9.3 (2021-04-22)
+=========================
+
+Features
+--------
+
+- Prevent the push key from being rejected for temporary errors and oversized payloads, add TTL logging, and support `events_only` push data flag. ([\#212](https://github.com/matrix-org/sygnal/issues/212))
+- WebPush: add support for Urgency and Topic header ([\#213](https://github.com/matrix-org/sygnal/issues/213))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing bug where invalid JSON would be accepted over the HTTP interfaces. ([\#216](https://github.com/matrix-org/sygnal/issues/216))
+- Limit the size of requests received from HTTP clients. ([\#220](https://github.com/matrix-org/sygnal/issues/220))
+
+
+Updates to the Docker image
+---------------------------
+
+- Remove manually added GeoTrust Root CA certificate from docker image as Apple is no longer using it. ([\#208](https://github.com/matrix-org/sygnal/issues/208))
+
+
+Improved Documentation
+----------------------
+
+- Make `CONTIBUTING.md` more explicit about how to get tests passing. ([\#188](https://github.com/matrix-org/sygnal/issues/188))
+- Update `CONTRIBUTING.md` to specify how to run code style and type checks with Tox, and add formatting to code block samples. ([\#193](https://github.com/matrix-org/sygnal/issues/193))
+- Document how to work around pip installation timeout errors. Contributed by Omar Mohamed. ([\#215](https://github.com/matrix-org/sygnal/issues/215))
+
+
+Internal Changes
+----------------
+
+- Update Tox to run in the installed version of Python (instead of specifying Python 3.7) and to consider specific paths and folders while running checks, instead of the whole repository (potentially including unwanted files and folders, e.g. the virtual environment). ([\#193](https://github.com/matrix-org/sygnal/issues/193))
+- Make development dependencies available as extras. Contributed by Hillery Shay. ([\#194](https://github.com/matrix-org/sygnal/issues/194))
+- Update `setup.py` to specify that a minimum version of Python greater or equal to 3.7 is required. Contributed by Tawanda Moyo. ([\#207](https://github.com/matrix-org/sygnal/issues/207))
+- Port CI checks to Github Actions. ([\#210](https://github.com/matrix-org/sygnal/issues/210), [\#219](https://github.com/matrix-org/sygnal/issues/219))
+- Upgrade development dependencies. Contributed by Omar Mohamed ([\#214](https://github.com/matrix-org/sygnal/issues/214))
+- Set up `coverage.py` to run in tox environment, and add html reports ([\#217](https://github.com/matrix-org/sygnal/issues/217))
+
+
+Sygnal v0.9.2 (2021-03-29)
+==========================
+
+Features
+--------
+
+- Add `allowed_endpoints` option as an understood config option for WebPush pushkins. ([\#184](https://github.com/matrix-org/sygnal/issues/184))
+- Add `ttl` option as an understood config option for WebPush pushkins to make delivery more reliable ([\#185](https://github.com/matrix-org/sygnal/issues/185))
+
+
+Sygnal 0.9.1 (2021-03-23)
+=========================
+
+Features
+--------
+
+- Add `allowed_endpoints` configuration option for limiting the endpoints that WebPush pushkins will contact. ([\#182](https://github.com/matrix-org/sygnal/issues/182))
+
+
+Bugfixes
+--------
+
+- Fix bug where the requests from different WebPush devices could bleed into each other. ([\#180](https://github.com/matrix-org/sygnal/issues/180))
+- Fix bug when using a HTTP proxy where connections would sometimes fail to establish. ([\#181](https://github.com/matrix-org/sygnal/issues/181))
+
+
+Sygnal 0.9.0 (2021-03-19)
+=========================
+
+Features
+--------
+
+- Add experimental support for WebPush pushkins. ([\#177](https://github.com/matrix-org/sygnal/issues/177))
+
+
+Bugfixes
+--------
+
+- Fix erroneous warning log line when setting the `max_connections` option in a GCM app config. ([\#157](https://github.com/matrix-org/sygnal/issues/157))
+- Fix bug where the `sygnal_inflight_request_limit_drop` metric would not appear in prometheus until requests were actually dropped. ([\#172](https://github.com/matrix-org/sygnal/issues/172))
+- Fix bug where Sygnal would not recover after losing connection to the database. ([\#179](https://github.com/matrix-org/sygnal/issues/179))
+
+
+Improved Documentation
+----------------------
+
+- Add preliminary documentation ([Troubleshooting](docs/troubleshooting.md) and [Application Developers' Notes](docs/applications.md)). ([\#150](https://github.com/matrix-org/sygnal/issues/150), [\#154](https://github.com/matrix-org/sygnal/issues/154), [\#158](https://github.com/matrix-org/sygnal/issues/158))
+- Add a note to the releasing doc asking people to inform EMS and customers during the release process. ([\#155](https://github.com/matrix-org/sygnal/issues/155))
+
+
+Internal Changes
+----------------
+
+- Remove a source of noisy (but otherwise harmless) exceptions in tests. ([\#149](https://github.com/matrix-org/sygnal/issues/149))
+- Add tests for HTTP Proxy support. ([\#151](https://github.com/matrix-org/sygnal/issues/151), [\#152](https://github.com/matrix-org/sygnal/issues/152))
+- Remove extraneous log line. ([\#174](https://github.com/matrix-org/sygnal/issues/174))
+- Fix type hints due to Twisted upgrade. ([\#178](https://github.com/matrix-org/sygnal/issues/178))
+
+
 Sygnal 0.8.2 (2020-08-06)
 =========================
 

CONTRIBUTING.md

@@ -1,13 +1,62 @@
 # Contributing code to Sygnal
 
-Everyone is welcome to contribute code to [matrix.org
-projects](https://github.com/matrix-org), provided that they are willing to
-license their contributions under the same license as the project itself. We
-follow a simple 'inbound=outbound' model for contributions: the act of
-submitting an 'inbound' contribution means that the contributor agrees to
-license the code under the same terms as the project's overall 'outbound'
-license - in our case, this is almost always Apache Software License v2 (see
-[LICENSE](LICENSE)).
+Everyone is welcome to contribute code to Sygnal, provided you are willing to
+license your contributions under the same license as the project itself. In
+this case, the [Apache Software License v2](LICENSE).
+
+## Preparing your development environment
+
+### Create a virtualenv
+
+To contribute to Sygnal, ensure you have Python 3.7 or newer and then run:
+
+```bash
+python3 -m venv venv
+./venv/bin/pip install -e '.[dev]'
+```
+
+This creates an isolated virtual Python environment ("virtualenv") just for
+use with Sygnal, then installs Sygnal along with its dependencies, and lastly
+installs a handful of useful tools
+
+If you get `ConnectTimeoutError`, this is caused by slow internet whereby 
+`pip` has a default time out of _15 sec_. You can specify a larger timeout 
+by passing `--timeout 120` to the `pip install` command above.
+
+Finally, activate the virtualenv by running:
+
+```bash
+source ./venv/bin/activate
+```
+
+Be sure to do this _every time_ you open a new terminal window for working on
+Sygnal. Activating the venv ensures that any Python commands you run (`pip`,
+`python`, etc.) use the versions inside your venv, and not your system Python.
+
+When you're done, you can close your terminal or run `deactivate` to disable
+the virtualenv.
+
+### Run the tests
+
+To make sure everything is working as expected, run the unit tests:
+
+```bash
+tox -e py
+```
+
+If you see a message like:
+
+```
+-------------------------------------------------------------------------------
+Ran 46 tests in 0.209s
+
+PASSED (successes=46)
+___________________________________ summary ___________________________________
+  py: commands succeeded
+  congratulations :)
+```
+
+Then all is well and you're ready to work!
 
 ## How to contribute
 
@@ -42,28 +91,24 @@ Sygnal follows the [Synapse code style].
 Many of the conventions are enforced by scripts which are run as part of the
 [continuous integration system](#continuous-integration-and-testing).
 
-To help check and fix adherence to the code style, you can run `scripts-dev/lint.sh`
-locally. You'll need Python 3.7 or later, and to install a number of tools:
-
-```
-# Install the dependencies
-pip install -U black flake8 isort mypy mypy-zope
+To help check and fix adherence to the code style, you can run `tox`
+locally. You'll need Python 3.7 or later, and a virtual environment configured and
+active:
 
-# Run the linter script
-./scripts-dev/lint.sh
-```
+```bash
+# Activate the virtual environment
+source ./venv/bin/activate
 
-**Note that the script does not just test/check, but also reformats code, so you
-may wish to ensure any new code is committed first**.
+# Run the code style check
+tox -e check_codestyle
 
-By default, this script checks all files and can take some time; if you alter
-only certain files, you might wish to specify paths as arguments to reduce the
-run-time:
-
-```
-./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
+# Run the types check
+tox -e check_types
 ```
 
+These commands will consider the paths and files related to the project (i.e.
+everything in `sygnal/` and in `tests/` as well as the `setup.py` file).
+
 Before pushing new changes, ensure they don't produce linting errors. Commit any
 files that were corrected.
 

README.rst

@@ -9,6 +9,12 @@ for a high level overview of how notifications work in Matrix.
 https://matrix.org/docs/spec/push_gateway/r0.1.0
 describes the protocol that Matrix Home Servers use to send notifications to Push Gateways such as Sygnal.
 
+
+Contributing
+============
+Looking to contribute to Sygnal? See `CONTRIBUTING.md <CONTRIBUTING.md>`_
+
+
 Setup
 =====
 Sygnal is configured through a YAML configuration file.
@@ -27,6 +33,7 @@ Keys in this section take the form of the ``app_id``, as specified when setting
 
 See the sample configuration for examples.
 
+
 App Types
 ---------
 There are two supported App Types:
@@ -65,6 +72,7 @@ gcm
   to contain the 'Server key', which can be acquired from Firebase Console at:
   ``https://console.firebase.google.com/project/<PROJECT NAME>/settings/cloudmessaging/``
 
+
 Using an HTTP Proxy for outbound traffic
 ----------------------------------------
 
@@ -78,6 +86,7 @@ Currently only HTTP proxies with the CONNECT method are supported.
 
 If you wish, you can instead configure a HTTP CONNECT proxy in ``sygnal.yaml``.
 
+
 Pusher ``data`` configuration
 =============================
 
@@ -90,6 +99,7 @@ via `POST /_matrix/client/r0/pushers/set <https://matrix.org/docs/spec/client_se
   This can be useful for clients to specify default push payload content. For instance, iOS clients will have
   freedom to use silent/mutable notifications and be able to set some default alert/sound/badge fields.
 
+
 Running
 =======
 
@@ -101,9 +111,20 @@ With custom configuration file name:
 
 Python 3.7 or higher is required.
 
+
 Log Rotation
 ============
+
 Sygnal's logging appends to files but does not use a rotating logger.
 The recommended configuration is therefore to use ``logrotate``.
 The log file will be automatically reopened if the log file changes, for example
 due to ``logrotate``.
+
+
+More Documentation
+==================
+
+More documentation for Sygnal is available in the ``docs`` directory:
+
+* `Notes for Application Developers <docs/applications.md>`_
+* `Troubleshooting <docs/troubleshooting.md>`_

RELEASING.md

@@ -1,3 +1,6 @@
+0. Consider whether this release will affect any customers, including those on
+EMS, and warn them beforehand - in case they need to upgrade quickly.
+
 1. Set a variable to the version number for convenience:
    ```sh
    ver=x.y.z
@@ -28,3 +31,5 @@
    ```sh
    xdg-open https://github.com/matrix-org/sygnal/releases/edit/v$ver
    ```
+1. Notify #sygnal:matrix.org, #synapse-dev:matrix.org and EMS that a new
+   release has been published.