Queue groups do not work across import/export and hub/leaf connections [v2.10.18 - v2.10.19]

[Karimerto]

Observed behavior

I am trying to set up a leaf account that can request specific services either from the local leaf node, or from the hub if a local service is not available. I configured two simple servers, hub and leaf, started request responders (with nats reply) and sent a request to the leaf node. What I see is the same request in both the hub-connected and leaf-connected responders. While it is indeed the leaf node that responds first, I was not expecting to even see a request on the hub node.

The hub user is different from the leaf user, and in different accounts (HubAccount and LeafAccount). I have exported the service from the hub user (HubUser) and imported it into the leaf user (LeafUser). I am using nats-resolver (and thus JWTs) for authentication, but only on the hub node (for now).

This is from the leaf node:

$ nats reply --server localhost:4333 'service.a.hello' leaf
11:44:07 Listening on "service.a.hello" in group "NATS-RPLY-22"
11:44:11 [#0] Received on subject "service.a.hello":

hello
^C11:44:22
Draining...
11:44:22 Exiting

This is from the hub node:

$ nats reply --server localhost:4222 --creds ~/nats/clustertest/test_nsc/creds/testop/HubAccount/HubUser.creds 'service.a.hello' hub
11:44:09 Listening on "service.a.hello" in group "NATS-RPLY-22"
11:44:11 [#0] Received on subject "service.a.hello":
11:44:11 Nats-Request-Info: {"acc":"ABI2WILJI5IR3EDLMZIGT3RSFDJEP5C5Y4TBYGMSJIPUGVQJAPCFFGLB","rtt":182422}

hello
11:44:23 [#1] Received on subject "service.a.hello":
11:44:23 Nats-Request-Info: {"acc":"ABI2WILJI5IR3EDLMZIGT3RSFDJEP5C5Y4TBYGMSJIPUGVQJAPCFFGLB","rtt":182422}

hello

I was not expecting to see the first request in the hub responder log seen above.

And here are the requests I sent:

$ nats req --server localhost:4333 'service.a.hello' 'hello'
11:44:11 Sending request on "service.a.hello"
11:44:11 Received with rtt 318.737µs
leaf


$ nats req --server localhost:4333 'service.a.hello' 'hello'
11:44:23 Sending request on "service.a.hello"
11:44:23 Received with rtt 583.934µs
hub

Expected behavior

As I mentioned, I expected that the hub node would not even see the first request. The service request should be served only from the leaf node as long as there is a responder available.

Server and client version

$ nats-server --version
nats-server: v2.10.18
$ nats --version
0.1.5

Host environment

This probably is not very relevant. I am running Ubuntu 22.04.4 LTS.

$ uname -a
Linux XXXXX 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce

Server config for hub node (hub.conf):

port: 4222
include resolver.conf
jetstream {
  store_dir: "./hub"
}
leafnodes {
  port: 7422
}

Server config for leaf node (leaf.conf):

port: 4333
jetstream {
  store_dir: "./leaf"
}
leafnodes {
  remotes: [
    {
      url: "nats-leaf://localhost",
      credentials: "~/nats/clustertest/test_nsc/creds/testop/LeafAccount/LeafUser.creds"
    }
  ]
}

Here are the steps to set everything up:

1. Initialize new nsc context:

   nsc init
   

2. Configure operator:

   nsc edit operator --account-jwt-server-url nats://localhost:4222
   nsc generate config --nats-resolver > resolver.conf
   

3. Start servers (in different terminals, and do note that leaf node cannot yet connect until changes are pushed):

   nats-server -c hub.conf
   nats-server -c leaf.conf
   

4. Add accounts and users:

   nsc add account LeafAccount --allow-pub "service.a.>" --allow-sub "_INBOX.>"
   nsc add user LeafUser --account LeafAccount
   nsc add account HubAccount
   nsc add user --account HubAccount HubUser
   

5. Add import/export (it does not matter whether you use a very specific subject, or either of the wildcards, they all fail in the same way):

   nsc add export --account HubAccount --service --subject "service.a.>"
   nsc add import --account LeafAccount --src-account HubAccount --remote-subject "service.a.>" --service
   

6. Push everything:

   nsc push -A
   

7. Start responders (in two more different terminals):

   nats reply --server localhost:4333 'service.a.hello' leaf
   nats reply --server localhost:4222 --creds ~/nats/clustertest/test_nsc/creds/testop/HubAccount/HubUser.creds 'service.a.hello' hub
   

8. Send a request:

   nats req --server localhost:4333 'service.a.hello' 'hello'
   

9. You should get a reply from the leaf node connected responder, but also see the same request appear on the hub node connected responder. This ought not happen though.

10. Kill the leaf node responder.

11. Send the same request as above, and correctly receive a reply from the hub node responder.

I have tried to read the documentation and figure out whether this is expected behavior, or whether I have some stupid configuration/permission issue somewhere but to no avail. Maybe this is a defect? Or expected behavior?

[Karimerto]

I tried this with the latest v2.10.19, but behavior is still the same.