[Bug]: Big file uploads chug every few hundreds of MB, Nextcloud client not respecting max chunk size

[plantroon]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When uploading a large file (say, 10 GB) to Nextcloud, the desktop client paues every few hundreds MB (not in a regular interval). Server load is high throughout the process. This has been occuring since I started using Nextcloud in August 2021 and no explanation was ever given for this problem until I investigated it myself today.

The corresponding setting on client-side is targetChunkUploadDuration (the default value of 6000 and dynamic chunk sizes is problematic) - this one enforces chunking no matter what I set on the server side - even if I set max chunk size to be 10 MB, the chunks created on the server-side look like this:

root@liberty:/var/lib/docker/volumes/nextcloud_nextcloud/_data/data/plantroon/uploads/2200454669# du -h *
9.6M    0000000000000000
331M    0000000000000001
518M    0000000000000002
602M    0000000000000003
647M    0000000000000004
671M    0000000000000005
155M    0000000000000006.ocTransferId580042766.part

This brings a low-end server to a high load situation and the client pauses every few hundred MBs.

Some background info: The disks being used are SSDs, the server is a Thinkpad X200 laptop.

I am reporting this as server issue, as it seems like the server's own limits are not enforced against the desktop client?

Steps to reproduce

1. Set up Nextcloud docker and the desktop client
2. Upload a large file
3. Watch the client's progress bar and server load and IO speeds to observe what it's doing
4. Use fatrace utility or anything else to further get insight into what's happening

Expected behavior

The client respects server's max chunk size or the upload is otherwise made smoother, without extreme load situation on server side.

Installation method

Official Docker image

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

No response

Are you using the Nextcloud Server Encryption module?

No response

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.plantroon.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "24.0.0.12",
        "overwriteprotocol": "https",
        "overwrite.cli.url": "http:\/\/nextcloud.plantroon.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": "true",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "loglevel": 0,
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - bookmarks: 10.3.1
  - bruteforcesettings: 2.4.0
  - cloud_federation_api: 1.7.0
  - dav: 1.22.0
  - federatedfilesharing: 1.14.0
  - files: 1.19.0
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_trashbin: 1.14.0
  - firstrunwizard: 2.13.0
  - lookup_server_connector: 1.12.0
  - nextcloud_announcements: 1.13.0
  - notifications: 2.12.0
  - oauth2: 1.12.0
  - password_policy: 1.14.0
  - phonetrack: 0.7.0
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - settings: 1.6.0
  - text: 3.5.1
  - twofactor_backupcodes: 1.13.0
  - updatenotification: 1.14.0
  - viewer: 1.8.0
  - workflowengine: 2.6.0
Disabled:
  - activity: 2.15.0
  - admin_audit
  - circles: 22.1.0
  - comments: 1.11.0
  - contactsinteraction: 1.2.0
  - dashboard: 7.1.0
  - encryption
  - federation: 1.11.0
  - files_external
  - files_sharing: 1.13.2
  - files_versions: 1.14.0
  - files_videoplayer: 1.11.0
  - logreader: 2.7.0
  - photos: 1.4.0
  - recommendations: 1.1.0
  - serverinfo: 1.12.0
  - sharebymail: 1.11.0
  - support: 1.5.0
  - survey_client: 1.10.0
  - systemtags: 1.11.0
  - theming: 1.12.0
  - user_ldap
  - user_status: 1.1.1
  - weather_status: 1.1.0
  - workflow_ocr: 1.23.2

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[rawtaz]

@github-actions Stop being silly. This issue should be automatically closed just because noone triaged it yet.. It's a legitimate issue, probably a bug, and needs to be investigated :)

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[rawtaz]

@plantroon @szaimen I and many others appreciate the work and time you put into NextCloud.

But seriously, what is the point of closing 100% legitimate issues, which also has very good information about the symptoms of the problem, without even having looked into them? If you (or the bot, rather) keep doing that, then what point is there to open an issue in the first place, when there's a bug or similar problem with NC?

Seriously, please configure your bot so that it doesn't automatically close issues like this. The way it works now is not only utterly stupid, but also highly counter-productive to improving the quality of the software.

[plantroon]

In the meantime I upgraded my server to one with AES acceleration which helped a little, but the bug report still stands as the Nextcloud client is not forced by the server to respect max chunk size (I think this should be fixed by the server to not allow malicious clients). If this worked correctly, the consequent issue with high server load wouldn't be as bad.

A note for the consequent high load issue: when uploading a 4 GB file, over 20 GB of data are written to disk on the server side. That's why it completely kills the performance, the file operations are just too heavy: client -> php upload tmp dir -> nextcloud data/user/upload dir -> assembling the chunks, done. Both apache and fpm version of docker image are affected. Btw, uploading a 4 GB file to the same server via sftp or even http post upload will only write 4 GB of data (obviously) and won't send the server's load avg through the roof xD

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[rawtaz]

@github-actions Well, what other details does your braindead software wish to receive?

@plantroon Not sure what "update the issue with new details" actually entails, but to be on the safe side, perhaps you should edit the initial post in this issue thread, adding some text just for kicks.

[plantroon]

@github-actions Well, what other details does your braindead software wish to receive?

@plantroon Not sure what "update the issue with new details" actually entails, but to be on the safe side, perhaps you should edit the initial post in this issue thread, adding some text just for kicks.

Checked with the latest updates (nextcloud 24) and updated the initial post. It is still happening. I do not test with different client version because this should simply not be allowed from server-side if it's set up to allow a certain max chunk size, no bigger size should be allowed in such case.

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[plantroon]

@github-actions still not resolved

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[plantroon]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

update: still not fixed ;(

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[rawtaz]

Here we go again. When will someone from Nextcloud even look at this bug report?

[szaimen]

there is this: #4826

[kesselb]

Hey 👋, thanks for your bug report 👍

The corresponding setting on client-side is targetChunkUploadDuration (the default value of 6000 and dynamic chunk sizes is problematic) - this one enforces chunking no matter what I set on the server side - even if I set max chunk size to be 10 MB, the chunks created on the server-side look like this:

https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html#adjust-chunk-size-on-nextcloud-side

You are referring to this part of the documentation right? I agree this gives the impression it would change the chunk size for the clients unfortunately it's only implemented in our web ui: https://github.com/nextcloud/server/search?q=max_chunk_size

As workaround for now I expect that the client configuration options targetChunkUploadDuration = 0 and chunkSize = 10000000 (https://github.com/nextcloud/desktop/blob/master/doc/conffile.rst) should make it work for you.

When #4826 is merged it should be possible to force the client to a given size by the webserver (e.g. client_max_body_size for nginx).

[plantroon]

Due to many changes on my server this is not a problem anymore, but thanks for letting me know about the possible fix in #4826 . Not exactly what I had in mind but it seems good enough.

I also realized that maybe this would have better been reported to server side, not to desktop client side.

I am fine with closing this issue at this point.