LFS: Client error when using multi-replica lfscache from invalid hmac signature

[aaronborden-rivian]

We're trying to configure lfscache in a high availability mode, using 2 or more replicas. One replica works just fine, but once a second is added, we see a git LFS Client Errors.

LFS: Client error: http://localhost:8000/_lfs_cache/77ec36d03f6209383a19162c683732df373d444b689698b50924d8815be485d4
LFS: Client error: http://localhost:8000/_lfs_cache/777c309f83d684467825cbc5ab81354fa62e455c70ad748fec80efa4af5ccb99

Sample logs:

12:04:08.710310 trace git-lfs: tq: starting transfer adapter "basic"
12:04:08.711549 trace git-lfs: HTTP: GET http://localhost:8000/_lfs_cache/6ae41d077e8556623c7cdcbaac1bab58058e247732c28d81e98e62d16bc6fd95
12:04:08.714053 trace git-lfs: HTTP: 200
12:04:08.715580 trace git-lfs: HTTP: GET http://localhost:8000/_lfs_cache/12ff0790a4e9c7cefb9d3a78feb13f94d8c70463344e4e2f2c7082f4bf6e231a
12:04:08.718284 trace git-lfs: HTTP: 400
12:04:08.719041 trace git-lfs: tq: refusing to retry "12ff0790a4e9c7cefb9d3a78feb13f94d8c70463344e4e2f2c7082f4bf6e231a", too many retries (8)
12:04:08.719088 trace git-lfs: tq: refusing to retry "12ff0790a4e9c7cefb9d3a78feb13f94d8c70463344e4e2f2c7082f4bf6e231a", too many retries (8)
12:04:08.720393 trace git-lfs: HTTP: GET http://localhost:8000/_lfs_cache/77ec36d03f6209383a19162c683732df373d444b689698b50924d8815be485d4
12:04:08.723638 trace git-lfs: HTTP: 400
12:04:08.724289 trace git-lfs: tq: refusing to retry "77ec36d03f6209383a19162c683732df373d444b689698b50924d8815be485d4", too many retries (8)
12:04:08.724324 trace git-lfs: tq: refusing to retry "77ec36d03f6209383a19162c683732df373d444b689698b50924d8815be485d4", too many retries (8)
12:04:08.725621 trace git-lfs: HTTP: GET http://localhost:8000/_lfs_cache/777c309f83d684467825cbc5ab81354fa62e455c70ad748fec80efa4af5ccb99
12:04:08.728185 trace git-lfs: HTTP: 400
12:04:08.728749 trace git-lfs: tq: refusing to retry "777c309f83d684467825cbc5ab81354fa62e455c70ad748fec80efa4af5ccb99", too many retries (8)
12:04:08.728761 trace git-lfs: tq: refusing to retry "777c309f83d684467825cbc5ab81354fa62e455c70ad748fec80efa4af5ccb99", too many retries (8)

The lfscache server produces no error logs, so it's difficult to track this down.

It looks like it's the signature verification from the X-Lfs-Signature header is failing. The hmac key is randomly generated for each instance, so a batch request served by one lfscache instance would not be valid against the other replicas.

I think being able to provide the hmac key as a CLI argument or environment variable would resolve this. Also, it would be nice if lfscache printed an error to the logs in this case.

[aaronborden-rivian]

If you have Docker Desktop available with Kubernetes, you can use these resource definitions to reproduce: https://gist.github.com/aaronborden-rivian/434132d8b2de4409f402cd8b69f084f9

kubectl apply -f .