Make Google API call in the background

[ben-z]

Currently, we execute up to 3 API calls in the /confirm endpoint:

1. get entity from azure table
2. create google group membership
3. delete entity from azure table

However, we've been running into SSLEOF errors when calling the Google API.

Internal reference: https://watonomous.sentry.io/issues/5728320814/?project=4507799285989376&query=is:unresolved+issue.priority:%5Bhigh,+medium%5D+!namespace:mimir+!namespace:wato-finance-frontend+!namespace:wato-finance-backend&statsPeriod=24h&stream_index=0

According to this, it's something to do with not having consistent CPU resources. Without digging into the details, we implement something that is general enough to handle this.

This PR offloads the Google API call to the background. Now, the /confirm endpoint simply:

1. Update azure table with a note saying that the entity is confirmed

And we run a background job called commit to do the rest:

1. query confirmed entities from azure table
2. create google group memberships
3. delete entities from azure table

The background job is idempotent, so we can retry it when it fails. We keep track of the last successful commit and healthcheck based on it.

During manual testing, if we trigger an exception in the background job loop, it will not interrupt the loop.

This PR changes the RowKey format and is not backwards compatible.

Tested and working in prod. It turns out the local instance of azure tables allows "eq null" (not sure if it works because "ne null" certainly doesn't work) and the cloud one doesn't. It has the error

HttpResponseError
One of the request inputs is not valid.
RequestId:5af5611d-e002-0045-48ee-f1c4ed000000
Time:2024-08-19T04:15:00.0135139Z
ErrorCode:InvalidInput

[ben-z]

Tested and working in prod. It turns out the local instance of azure tables allows "eq null" (not sure if it works because "ne null" certainly doesn't work) and the cloud one doesn't. It has the error

HttpResponseError
One of the request inputs is not valid.
RequestId:5af5611d-e002-0045-48ee-f1c4ed000000
Time:2024-08-19T04:15:00.0135139Z
ErrorCode:InvalidInput