Bulk upload issues

[joshtrichards]

(WIP: will be updated as needed)

Summary: Indications are that there are one or more scenarios that trigger problems with bulk uploads, creating serious performance problems (e.g. 100% CPU usage, non-responsive). Many find success through workarounds like disabling bulk uploads or adjusting upload limits. Some reporters have noted that #5680 resolved their situation, while others it has not. There may either be additional triggers or simply multiple unrelated root causes.

Note: Many of these are likely duplicates (which is part of what needs to get sorted out)

Issues:

• [Bug]: after upgrade to >=3.8.2 the client consumes 100% CPU and shows "synchronising" when opened from the tray #5726
  • OP reports fixed (by Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680): [Bug]: after upgrade to >=3.8.2 the client consumes 100% CPU and shows "synchronising" when opened from the tray #5726 (comment))
    • Should likely be closed as a result; new reporters clearly have different circumstances so should create new Issue(s) as needed
• [Bug]: Desktop Client freezes in a 100% CPU load loop after server upgrade to 25 #5094
  • OP reports fixed (by Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680): [Bug]: Desktop Client freezes in a 100% CPU load loop after server upgrade to 25 #5094 (comment)
    • OP reports still fixed: [Bug]: Desktop Client freezes in a 100% CPU load loop after server upgrade to 25 #5094 (comment)
    • Should stay closed as a result; new reporters clearly have different circumstances so should create new Issue(s) as needed
• [Bug]: Windows Desktop Client UI completely unresponsive during preparation for sync #5031
• [Bug]: Unknown error and network error 99 when syncing .vscode extensions #5349
• Windows, Linux, and macOS clients freezing when bulk uploading #4106
  • OP reports fixed (by Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680): Windows, Linux, and macOS clients freezing when bulk uploading #4106 (comment)
• [Bug]: Windows Client Error on Every File Creation #5888
• [Bug]: Computed md5 hash is incorrect #6511
• Bulk Upload Issues server#29702
• [Bug]: InvalidArgumentException: X-OC-MTime header must be a valid positive integer (regression?) & The X-File-MD5 header must not be null.  #6889
• [Bug]: Please re-open issue 5094, bulk upload still triggers 100%CPU client usage and hangs  #6941

PRs:

• Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680
• https://github.com/nextcloud/desktop/pulls?q=bulk+is%3Apr+is%3Aclosed+

[sapphirepro]

Yeah, when uploading bulk of files, it wastes cpu and after a tiny upload goes rescan etc in a loop. With fiber network uploading like 1 GB of photos can take even an hour to complete. So solution still needed

[mgallien]

thanks for the help raising the importance of this
is there known reproducible scenario that are still failing with recent server and client releases ?
no big hopes of a quick fix if I first need to investigate and spend hours trying to analyze possible ways to trigger the bugs

[sapphirepro]

thanks for the help raising the importance of this is there known reproducible scenario that are still failing with recent server and client releases ? no big hopes of a quick fix if I first need to investigate and spend hours trying to analyze possible ways to trigger the bugs

Ok. Giving information. Latest NextCloud stable release. Latest Linux desktop client. Now situation is that to client side dropped 700 mb of files (approx 1-1.5 Mb per file, photos). So key scenario to test reasoanble size like 700 MB or above, while they should be small and a lot of them, like 100 files or above. Sync will be like dropping fast maybe 30-40 mb, then huuuge wait like few minutes, then again short time upload of 30-40 mb and so on in the loop till all uploaded.

Expected behavious is if dropped 700mb of many files to upload, then upload them all non stop and only then resync. What I suspect as a reason of this performance bug, is that it starts upload, uploads tiny files like 40 mb in total and then server reports changes (those same files), upload gets held, resync and then proceed. It should throw all files in a bunch and only then resync something. Also a tray icon and clicking on it while such sync doesn't often show floating window of nextcloud desktop client, by which I assume that there is process violation as well, that sync or something being done in main process and blocking UI, while it should be a separate thread.

I tried the best I could to explain. If any more info needed, do tell, I will try to assist.

[Bockeman]

My Windows10 Nextcloud client is 3.13.2 (Stable).
I ugraded my (Fedora 40)server to Nextcloud Hub 8 (30.0.0 RC1) (Beta and soon to be RC2).
Consequently, I got loads of errors and warnings which I have largely eliminated apart from "Error no app in context Computed md5 hash is incorrect" which appears to be related to this topic.

I have "'bulkupload.enabled' => false," in my server nextcloud.cfg (though I am not convinced this is effective).

My Windows10 Nextcloud client settings shows 20TB in use, and this includes Sync Connections for AppData/Roaming. The important point being that files are changing constantly at the whim of Windows.

Bulk transfer seems a good option IMHO, but there is a high likelyhood of file changes before the transfer is complete. Hence I am expecting all files uploaded to the server to retain the file modified date, as captured in the bulk, and a subsequent transfer to handle files changed since the previous bulk capture.

I am getting "Error no app in context Computed md5 hash is incorrect" every 10 minutes or so, therefore I think I have a suitable environment for testing possible fixes/patches.

Please advise if there is any patch you wish me to apply to verify possible solutions.