Add additional bindings to filter end devices and gateways

[halimi]

Summary

We introduced a new filter field in ListGatewaysRequest and ListEndDevicesRequest RPCs however this filter only works on grpc endpoint and not on the http endpoint. The filter is an embedded field in the request and looks like this.

  message Filter {
    oneof field {
      google.protobuf.Timestamp updated_since = 1 [(validate.rules).timestamp.lt_now = true];
    }
  }
  repeated Filter filters = 6;

The HTTP API generator doesn't recognize it as a query parameter therefore not possible to use it in a HTTP request.
https://github.com/TheThingsNetwork/lorawan-stack/blob/v3.32/api/ttn/lorawan/v3/api.swagger.json#L1052-L1089

Changes

Added an additional HTTP binding on the List RPC call. This new binding allows a POST request where the filter field can be used in the payload.

  rpc List(ListEndDevicesRequest) returns (EndDevices) {
    option (google.api.http) = {
      get: "/applications/{application_ids.application_id}/devices"
      additional_bindings {
        post: "/applications/{application_ids.application_id}/devices/filter"
        body: "*"
      }
    };
  }

Testing

Steps

• List the gateways without filter using a GET HTTP request on http://localhost:1885/api/v3/gateways
• List the gateways with updated_since filter using a POST method on http://localhost:1885/api/v3/gateways/filter
• List the end devices without filter using a GET method on http://localhost:1885/api/v3/applications/{application id}/devices
• List the end devices with updated_since filter using a POST method on http://localhost:1885/api/v3/applications/{application id}/devices/filter

Results

• List the gateways without filter using a GET HTTP request on http://localhost:1885/api/v3/gateways

curl -H "Accept: application/json" -H "Authorization: Bearer $TOKEN"  http://localhost:1885/api/v3/gateways
{"gateways":[{"ids":{"gateway_id":"gw1","eui":"1133225544667788"},"created_at":"2024-08-26T15:33:26.530917Z","updated_at":"2024-08-26T15:33:26.530918Z"}]}%

• List the gateways with updated_since filter using a POST method on http://localhost:1885/api/v3/gateways/filter

curl -X POST -H "Accept: application/json" -H "Authorization: Bearer $TOKEN" -d "@req.txt"  http://localhost:1885/api/v3/gateways/filter
{"gateways":[{"ids":{"gateway_id":"gw1","eui":"1133225544667788"},"created_at":"2024-08-26T15:33:26.530917Z","updated_at":"2024-08-26T15:33:26.530918Z"}]}%

cat req.txt
{
  "field_mask": {
    "paths": ["name"]
  },
  "filters": [
    {
      "updated_since": "2024-08-26T15:33:26.530918Z"
    }
  ]
}

• List the end devices without filter using a GET method on http://localhost:1885/api/v3/applications/{application id}/devices

curl -H "Accept: application/json" -H "Authorization: Bearer $TOKEN"  http://localhost:1885/api/v3/applications/my-new-application/devices
{"end_devices":[{"ids":{"device_id":"dev11","application_ids":{"application_id":"my-new-application"},"dev_eui":"1122335544667788","join_eui":"0000000000000000"},"created_at":"2024-08-27T15:54:29.800642Z","updated_at":"2024-08-27T15:54:29.800643Z"},{"ids":{"device_id":"dev1111","application_ids":{"application_id":"my-new-application"},"dev_eui":"0004A30B001C0530","join_eui":"800000000000000C"},"created_at":"2024-09-02T15:34:12.237313Z","updated_at":"2024-09-02T15:34:12.237314Z"},{"ids":{"device_id":"dev122222","application_ids":{"application_id":"my-new-application"},"dev_eui":"4488775533221166","join_eui":"2244668811335577"},"created_at":"2024-09-04T08:05:57.600464Z","updated_at":"2024-09-04T08:05:57.600465Z"},{"ids":{"device_id":"eui-0004a310001ab132","application_ids":{"application_id":"my-new-application"},"dev_eui":"0004A310001AB132","join_eui":"70B3D57ED0000000"},"created_at":"2024-06-13T09:22:45.008Z","updated_at":"2024-06-13T09:22:45.008Z"},{"ids":{"device_id":"eui-8877665544332211","application_ids":{"application_id":"my-new-application"},"dev_eui":"8877665544332211","join_eui":"1122334455667788"},"created_at":"2024-06-12T09:39:32.802981Z","updated_at":"2024-06-12T09:39:32.802983Z"},{"ids":{"device_id":"eui-fedcba9876543211","application_ids":{"application_id":"my-new-application"},"dev_eui":"FEDCBA9876543211","join_eui":"123456789ABCDEF1"},"created_at":"2024-06-12T09:34:39.899050Z","updated_at":"2024-06-12T09:34:39.899051Z"}]}%

• List the end devices with updated_since filter using a POST method on http://localhost:1885/api/v3/applications/{application id}/devices/filter

curl -X POST -H "Accept: application/json" -H "Authorization: Bearer $TOKEN" --data "@req2.txt"  http://localhost:1885/api/v3/applications/my-new-application/devices/filter
{"end_devices":[{"ids":{"device_id":"dev122222","application_ids":{"application_id":"my-new-application"},"dev_eui":"4488775533221166","join_eui":"2244668811335577"},"created_at":"2024-09-04T08:05:57.600464Z","updated_at":"2024-09-04T08:05:57.600465Z"}]}%

cat req2.txt
{
  "application_ids": {
    "application_id": "my-new-application"
  },
  "field_mask": {
    "paths": ["name"]
  },
  "filters": [
    {
      "updated_since": "2024-09-04T08:05:57.600465Z"
    }
  ]
}

Regressions

...

Notes for Reviewers

@johanstokking The filter is an embedded repeated field in the request, probably that is why not recognizing the generator as a query parameter. Together with @KrishnaIyer we came up with the idea to use and additional binding on the HTTP endpoint. It is a POST method where the filter field can be used in the payload. It works but the only drawback is it is adding a POST method to a List endpoint besides the default GET method. @johanstokking what do you think? Do you know a better solution?

Checklist

• Scope: The referenced issue is addressed, there are no unrelated changes.
• Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md for the chosen target branch.
• Documentation: Relevant documentation is added or updated.
• Testing: The steps/process to test this feature are clearly explained including testing for regressions.
• Infrastructure: If infrastructural changes (e.g., new RPC, configuration) are needed, a separate issue is created in the infrastructural repositories.
• Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
• Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.

[johanstokking]

This is the right way to do it indeed. It is more important that we allow for more complex filtering than to support GET with path and query parameters.