Connection fails when server sends compressed responses

[jpmcm1]

⚠️ Before posting ⚠️

• This is a bug, not a question or an enhancement.
• I've searched for similar issues and didn't find a duplicate.
• I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
• I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

0. All devices are connected in the same Tailscale net.

1. Create a standard Nextcloud instance with docker.

2. Reverse-proxy this instance through a second docker Nginx container.

3. Generate a Tailscale HTTPS certificate, and apply to nginx proxy config.

4. Run the instance!

5. Connect to Nextcloud with another device on the tailscale network, ie: cloud.tailxxxxx.ts.net. Complete setup.

6. Connect Android device to Tailscale.

7. Try to connect with Android client through a Tailnet address (like cloud.tailxxxxx.ts.net). FAILS with "Server not found"

Expected behaviour

Android client should connect.

Actual behaviour

Client fails with "Server not found".

Note that this is very much an Android-client only issue! ON the exact same Android phone, without changing any connection, I can load the nextcloud page through firefox without any trouble. I also have tested this on a Macbook and the ipad app from the app store, all of which also work fine.

Android version

13

Device brand and model

Samsung S21+

Stock or custom OS?

Stock

Nextcloud android app version

Nextcloud-android/20221227 (dev)

Nextcloud server version

25.0.2

Using a reverse proxy?

Yes

Android logs

Abbreviated log; my tailnet info, etc. changed.

12-31 00:31:48.980 14616 14616 D InputMethodManager: startInputInner - Id : 0
12-31 00:31:48.980 14616 14616 I InputMethodManager: startInputInner - mService.startInputOrWindowGainedFocus
12-31 00:31:48.982 14616 14616 D OperationsService: Starting command with id 22
12-31 00:31:48.983 14616 14665 D OwnCloudClient #0: REQUEST GET /status.php
12-31 00:31:48.986 14616 14665 D AdvancedSslSocketFactory: Creating SSL Socket with remote <MY.TAILNET>.ts.net:443, local null:0, params: org.apache.commons.httpclient.params.HttpConnectionParams@e05850a
12-31 00:31:48.986 14616 14665 D AdvancedSslSocketFactory:  ... with connection timeout 50000 and socket timeout 50000
12-31 00:31:48.991 14616 14665 D TrafficStats: tagSocket(153) with statsTag=0xffffffff, statsUid=-1
12-31 00:31:48.991 14616 14665 I ServerNameIndicator: SSLSocket implementation: org.conscrypt.Java8EngineSocket
12-31 00:31:48.991 14616 14665 I ServerNameIndicator: SNI done, hostname: <MY.TAILNET>.ts.net
12-31 00:31:49.092 14616 14665 W HttpMethodBase: Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
12-31 00:31:49.092 14616 14665 E GetStatusRemoteOperation: Connection check at https://<MY.TAILNET>.ts.net: The Nextcloud server is not configured!
12-31 00:31:49.092 14616 14665 D GetStatusRemoteOperation: establishing secure connection failed, trying non secure connection
12-31 00:31:49.092 14616 14665 D OwnCloudClient #0: REQUEST GET /status.php
12-31 00:31:49.092 14616 14665 D TrafficStats: tagSocket(81) with statsTag=0xffffffff, statsUid=-1
12-31 00:31:49.106 14616 14665 E GetStatusRemoteOperation: Connection check at http://<MY.TAILNET>.ts.net: Local file does not exist
12-31 00:31:49.106 14616 14665 D OperationsService: Called 1 listeners
12-31 00:31:49.106 14616 14665 D OperationsService: Stopping after command with id 22
12-31 00:33:25.547 14616 14616 I ViewRootImpl@b620148[AuthenticatorActivity]: ViewPostIme pointer 0
12-31 00:33:25.592 14616 14616 I ViewRootImpl@b620148[AuthenticatorActivity]: ViewPostIme pointer 1

Server error logs

<SERVER IP> - - [31/Dec/2022:00:39:55 +1100] "GET /status.php HTTP/1.1" 200 151 "-" "Mozilla/5.0 (Android) Nextcloud-android/20221227"

Additional information

The server is definitely seeing the Android request status.php, though I'm unsure why it's insisting the server isn't configured.

I can get the Android client to work IF I point it directly at the internal (not tailscale) dns name.

[AlvaroBrey]

Can you paste the result of status.php? if possible, curl it through both routes (vpn and internal ip) to see if there's any difference.

curl -X GET http://<your server address>/status.php

[jpmcm1]

Hi sorry for the wait! This is using an internal ip:

% curl -X GET https://..../status.php
{"installed":true,"maintenance":false,"needsDbUpgrade":false,"version":"25.0.3.2","versionstring":"25.0.3","edition":"","productname":"Nextcloud","extendedSupport":false}

while going through the external gives:

% curl  --compressed -X GET https://.......ts.net/status.php
{"installed":true,"maintenance":false,"needsDbUpgrade":false,"version":"25.0.3.2","versionstring":"25.0.3","edition":"","productname":"Nextcloud","extendedSupport":false}

I did have to add the --compressed flag to the curl for this one, else it would return binary!

[AlvaroBrey]

I did have to add the --compressed flag to the curl for this one, else it would return binary!

Hm, I'm not entirely familiar with how curl deals with this kind of stuff. Is it possible that the response is compressed but doesn't contain compression headers? That could likely cause problems with the HTTP client in the app. Please post both curl results again, but this time use curl -v so we can see the headers.

Additionally, is there a possibility that you may give me a test account for your server? If so, please send the credentials to alvaro.brey AT nextcloud.com along with a link to this issue.

[jpmcm1]

Hi Alvaro, here's the header output:

% curl -v  -X GET https://xxxxxx.ts.net/status.php
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying x.x.x.x:443...
* Connected to xxxxxx.ts.net (x.x.x.x) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /opt/anaconda3/ssl/cacert.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=xxxxxx.ts.net
*  start date: Dec 30 06:50:26 2022 GMT
*  expire date: Mar 30 06:50:25 2023 GMT
*  subjectAltName: host "xxxxxx.ts.net" matched cert's "xxxxxx.ts.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /status.php]
* h2h3 [:scheme: https]
* h2h3 [:authority: xxxxxx.ts.net]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x13c013c00)
> GET /status.php HTTP/2
> Host: xxxxxx.ts.net
> user-agent: curl/7.84.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Fri, 20 Jan 2023 00:26:12 GMT
< content-type: application/json
< vary: Accept-Encoding
< set-cookie: oc_sessionPassphrase=OjDUfJCBBHMykBKfbbScmCyBslA%2F07tnQJLR1delPG1%2FmxbkgkwPj9ks0vnlxrMy1nQb2IY%2BW%2F0PnXsq%2BcDzreGnE0tw3kRlo%2FK%2B4faziPg4P2GIH45VoJ6qtWhe2ggq; path=/; secure; HttpOnly; SameSite=Lax
< content-security-policy: default-src 'self'; script-src 'self' 'nonce-QThaQjRMSzhwVVhDTVd4RVpzNHl5K2RwRklKUnFpNHlYVThvUEVBYTg4RT06YlkwSGxJSFI3aTJ3QXdZaVVJRjBtNkpjUXN3aHlFQkFLejVnVXpRaXVJOD0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
< set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< set-cookie: ocqk2bjfr6og=7vi56armjh6m248s1de1etsk92; path=/; secure; HttpOnly; SameSite=Lax
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< cache-control: no-store, no-cache, must-revalidate
< pragma: no-cache
< access-control-allow-origin: *
< referrer-policy: no-referrer
< x-download-options: noopen
< x-permitted-cross-domain-policies: none
< x-robots-tag: none
< x-xss-protection: 1; mode=block
< content-encoding: gzip
< front-end-https: on
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains
< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Connection #0 to host xxxxxx.ts.net left intact

I did some debugging with the app itself too the other night; it's definitely an issue based on if the server sends the status.php response in binary form.

From the call RemoteOperationResult result = getStatus.execute(client); (GetServerInfoOperation.java, line 80) ultimately leads to GetStatusRemoteOperation.class (line 65): there is a call to getResponseBodyAsString(), followed by a conversion into a JSON. This will always fail, and that's why the return value suggests the server isn't actually setup yet.

Since from what I can tell none of the other clients (either Win or ios/osx) actually do this request to status.php, they have no issues connecting -- this is probably why only the Android client is showing this problem!

[AlvaroBrey]

< content-encoding: gzip

So the headers are correct. Not sure why our HTTP lib is not picking this up or if it just doesn't support it... but if it was that case, I'm perplexed that this didn't come up earlier.

Some technical context: the library uses two different HTTP clients: Apache HttpClient, which is the source of a lot of problems, and OkHttp. This operation still uses the old client, and switching it to OkHttp should fix it... but you'll probably end up with the same problem in a different operation.

Since we don't have the time to completely port all operations to OkHttp now (although some progress is being made (nextcloud/android-library#881), I'd suggest for you as a workaround to disable compression in Tailscale if you can. I know it's not a good workaround, but it is a workaround anyway.

I'll try to set up a local proxy with compression just to ensure that this is what's happening.

[jpmcm1]

Hey thanks again for taking a look at this! Very interesting.. I might bounce this thread over to the Tailscale guys and see if they have any thoughts -- the compressed response might be something to do with the way they forward the Wireguard connection.

[joshtrichards]

A thought: retry the curl with --http1.1 specified. That ancient HttpClient @AlvaroBrey mentioned only supports HTTP/1.1...

So the headers are correct. Not sure why our HTTP lib is not picking this up or if it just doesn't support it... but if it was that case, I'm perplexed that this didn't come up earlier.

AFAIK HttpClient 3.1 doesn't support gzip content encoding.

Though I too can't for the life of me fathom this not coming up before...

Maybe there's a server in the path here that is very insistent about gzip...

So for curl testing it might also be worth throwing an explicit Accept-Encoding: identity in there for good measure...

Also, question:

I can get the Android client to work IF I point it directly at the internal (not tailscale) dns name.

Does this test (which works) also bypass your reverse proxy too then or am I misunderstanding the topology?

[jpmcm1]

Hey all, with a fresh install of nextcloud 28, and the latest nextcloud/tailscale apps, I can confirm this no longer throws an error.