File Client Inacessible - Tailscale Docker Compose

[Servinjesus1]

I have endeavored to learn how to self-host and feel extremely close, but it feels like one last thing isn't working.

I am running my test node using the default docker compose repo with EXTERNAL_LISTEN_HOSTS="<tailscale>.ts.net 127.0.0.1" so locally, the node is accessible and client.yml is populated with the externally accessible Tailscale address. Then, I have to use Tailscale funnel to broadcast ports:

tailscale funnel --bg --https 443 --set-path /$ANY_SYNC_NODE_1_HOST $ANY_SYNC_NODE_1_PORT;
tailscale funnel --bg --https 443 --set-path /$ANY_SYNC_NODE_2_HOST $ANY_SYNC_NODE_2_PORT;
tailscale funnel --bg --https 443 --set-path /$ANY_SYNC_NODE_3_HOST $ANY_SYNC_NODE_3_PORT;
tailscale funnel --bg --https 443 --set-path /$ANY_SYNC_COORDINATOR_HOST $ANY_SYNC_COORDINATOR_PORT;
tailscale funnel --bg --https 443 --set-path /$ANY_SYNC_FILENODE_HOST $ANY_SYNC_FILENODE_PORT;
tailscale funnel --bg --https 443 --set-path /$ANY_SYNC_CONSENSUSNODE_HOST $ANY_SYNC_CONSENSUSNODE_PORT;

This establishes proxies to the local nodes with open ports:

# Funnel on:
#     - https://<tailscale>.ts.net

https://<tailscale>.ts.net (Funnel on)
|-- /any-sync-node-2        proxy http://127.0.0.1:1002
|-- /any-sync-node-3        proxy http://127.0.0.1:1003
|-- /any-sync-node-1        proxy http://127.0.0.1:1001
|-- /any-sync-filenode      proxy http://127.0.0.1:1005
|-- /any-sync-coordinator   proxy http://127.0.0.1:1004
|-- /any-sync-consensusnode proxy http://127.0.0.1:1006

which are accessible on other computers via curl -v https://<tailscale>.ts.net/any-sync-node-1:

*   Trying <IP>:443...
* TCP_NODELAY set
* Connected to <tailscale>.ts.net (<IP>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=<tailscale>.ts.net
*  start date: Jul 15 18:17:25 2024 GMT
*  expire date: Oct 13 18:17:24 2024 GMT
*  subjectAltName: host "<tailscale>.ts.net" matched cert's "<tailscale>.ts.net"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55d2df6ca0e0)
> GET /any-sync-node-1 HTTP/2
> Host: <tailscale>.ts.net
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502 
< content-length: 0
< date: Thu, 18 Jul 2024 00:45:10 GMT
< 
* Connection #0 to host <tailscale>.ts.net left intact

To make this compatible with AnyType, I thought to modify my client.yml as:

nodes:
  - addresses:
      - any-sync-node-1:1001
      - quic://any-sync-node-1:1011
      - <tailscale>.ts.net:443/any-sync-node-1
      - quic://<tailscale>.ts.net:1011
      - 127.0.0.1:1001
      - quic://127.0.0.1:1011
    peerId: ...

i.e. replacing the local port with tailscale's listen port and path.

netcheck tells me that at least for the coordinator node, this is unsuccessful (unknown port).

2024-07-18T00:49:32.246Z        WARN    netcheck        open TCP conn error     {"addr": "<tailscale>.ts.net:443/any-sync-coordinator", "error": "dial tcp: lookup tcp/443/any-sync-coordinator: unknown port", "dur": "1.194967ms"}

Which seems to indicate that maybe non-root paths aren't supported. This would be unfortunate because Tailscale funneling can only listen on 3 ports (443, 4430, and 10000). So, to expose all nodes, I'd need at least two tailscale sockets.

---

If anyone has managed to get this working using Tailscale or a similarly straightforward service for someone behind a NAT, I'd appreciate your help. Further, am I right in understanding that the sync, coordinator, file, and consensus nodes must all be exposed for external clients to synchronize? I've tried with just the coordinator node and that didn't work so I worried that external clients need all nodes per the sync protocol.

[fb929]

1. "<tailscale>.ts.net:443/any-sync-node-1" - this looks like an HTTP reverse proxy.
   our application does NOT use the HTTP protocol! we use gRPC and QUIC.
   therefore, the port should be forwarded through NAT or another method that ensures the correct forwarding of TCP and UDP ports.
2. currently, "netcheck" can only verify the availability of coordinators.
   but for proper functionality, you need to ensure the availability of all 'any-sync-*' daemons
3. tailscale prohibits mDNS, so peer-to-peer synchronization will not work; only synchronization through the server will be possible

[Servinjesus1]

OK. Alas, I saw this and couldn't decipher the acronym soup on first glance, but I guess it means Tailscale doesn't support QUIC (at least not UDP).

Claude has suggested ZeroTier to me, so I'll look into that. I am very much a newbie to self hosting and a cheapskate, trying to set this up for my academic research group. Really hope this works because a self-hosted relational database would make an exceptionally powerful electronic lab notebook and wiki for the team!