[Bug]: Nextcloud client generates >100 GB worth of log files every day (800MB per sync run). #5302
Comments
|
Related: #3312 |
|
A temporary fix could be adding a radio button to enable/disable logging until verbosity can be reduced only to errors. |
|
On Linux I've literally set the logs folder to read-only, I've tried everything and nothing helped. I was working and suddenly my SSD ran out of space and the editor was closed abruptly because it failed to save its data. Nextcloud generated over 200GB of logs and the directory was full of compressed log files and a single text file with more than 150GB+. I've then traced the problem to Dolphin, but Dolphin was not the culprit, it was just requesting file status for every file in my I don't know if this will also work for Windows (maybe Nextcloud can just try to change the permission on Windows, on Linux it's not possible without root privileges), but you can try as well. The only downside is that you lose all logs, but it's better than losing an NVMe SSD because of the massive wear. |
|
hmm, I use the appimage (previously a repo as well) on Ubuntu 20 (for years now) and have never had log file issues. |
|
Iteresting. Just out of curiousity, did you check the size of the logs folder? What size is it?
|
|
Oh I just remembered, there is no virtual file support for linux (yet, I hope), so maybe there is the culprit, and this issue only impacts Windows users using the virtual file support. |
|
We should have an option to reduce log level in the client. My But I still have hundreds of megabytes in compressed log files! |
|
I have this issue also, although I didn't enable virtual files yet, the log folder ballooning to hundreds of gigabytes! I am on Windows 10, just reinstalled the client to confirm the issue (now 3.8.1) My data is around 25GB, ~50.000 files. I had to symlink the log folder to a spinning HD to move it, but that is not a solution because its growing too fast. I really hope this will be fixed soon, I am surprised that you don't have more reports on this. |
|
Same here - version 3.7.4 has 239MBs of zipped logs from one day only and I don't have as many files (~1TB / 100k files on server) - and I didn't made any excessive changes in the last days.. In my case there are sets of 4 files every 2 hours.. where archives of the same size are created
30MB archive is ~600MB extracted and it looks like it contains some log records to literally each file stored in my account.. even it didn't change at this time.. on sample of random directory record : another sample is for a file: take a look at UPDATE: adding doesn't change anything.. still 30MB archive (600MB plain log) is written every 2 hours :( |
|
hopefully the problem is improved due to #5634 UPDATE:
UPDATE 2:
I think if the log output of @allexzander as you are working on useless logs already would be great you can look at this issue as well. |
|
The number of log files is growing at an alarming rate. We need to do something about it immediately. |
|
I'm not very sure about the size of the logs.. I 'm under impression the size is reduced with the last version but there was a huge user experience regression - now an archive is generated for every 512K log - resulting in insane number of of log archives.. my first sync of the day consists of 1291 files (33Mb).. two hours later same files count/size.. I don't think it changes the whole size of logging but makes it harder to follow/understand. definitely killing useless logs by default is a MUST. EDIT: another point is definitely personal data included in the logs. current implementation leaks all file names stored in the account through the logs. The warning introduced with #6237 is more like a joke if the user is expected to anonymize gigabytes of logs. |
|
I have also seen thousands of gzipped nextcloud log files, lucky for me I have reserved a folder in my ramdisk for these files (my ssd thank me for it). |
|
for me update to 3.9.0 reduces logging from 30MB archives every 2h to 20MB (likely due to #5796) not many but a right direction already. |
|
I created a task in Task Scheduler that deletes all |
|
I just tried installing the Windows client again, now in version 3.9.0 - in only 1 hour I have 265MB of log files, logging every small action - this is totally weird behaviour and makes Nextcloud unusable for me, also - I can't recommend it to anyone in this state. |
|
Same here. For me it looks like the client keeps syncing in a loop. Every attempt a new logfile is created. A single attempt takes only a second till the next attempt is started (I can see the green icon in client turns quickly to the blue icon during the second of sync attempt). 2023-09-16 03:27:42:735 [ info nextcloud.gui.socketapi C:\Users\sysadmin\AppData\Local\Temp\2\windows-17553\client-building\desktop\src\gui\socketapi\socketapi.cpp:338 ]: Lost connection QLocalSocket(0x21764436660) Being a Nextcloud user since many years now, my impression is that the Nextcloud client is tested very badly. Often there have been issues after updates. Problems with data being deleted, corrupted or now endlessly logged. Not good at all! Win10 x64 |
|
I hoped 3.10 will fix the the issue due to: "Bugfix/adjust log levels by @mgallien in" #5796 but still huge logs - in my case this are around zipped 20MB (~250MB) logs on each client start and awake from sleep.. now the logs lines look like this - including very strange static string '[ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-17670\client-building\desktop\src\libsync\discovery.cpp:464 ]' @mgallien - can we please got it sorted? there is no value of logging each existing file.. normal logs should only log when files are changed and require replication to or from the server. |
|
I feel like it is overlooked, that most of the users don't even need the logs... I'm pretty sure, most don't even know they exist. Can't there just be a switch to turn logging on/off? It should be relatively easy to implement, and it would solve the issue. What speaks against it? |
Didn't look like a priority #3312 |
|
I find it infuriating that if I set the log folder to read only Nextcloud keeps popping up a message (and switching the window focus to it) every few seconds saying that the log folder isn't writable. |
|
I also have this issue, it even persists in Nextcloud-3.11.0-x64.msi. 2 suggestions:
I had to delete the log-folder each day using manually or my partition runs out of space. |
|
BTW at least for Nextcloud 3.11 on Debian setting an environment variable QT_LOGGING_RULES for the Nextcloud process seems to help somewhat. To do that, you can copy /usr/share/dbus-1/services/com.nextcloudgmbh.Nextcloud.service to ~/.local/share/dbus-1/services/ (keep the filename) and /usr/share/applications/com.nextcloud.desktopclient.nextcloud.desktop to ~/.local/share/applications/ and then, in both copied files, adjust the |
|
I hope this gets fixed soon. For years this habit is wearing off my SSD. An alternative workaround would also be to sync via rsync or so, but this is a bit fiddly. |
|
Currently facing the same issue with v3.13.2 (on Linux) How has still not been solved, or at least been improved ? Maybe I'm missing something |
Yes, I find it infuriating too. I had once spent some time trying to build a custom fork of client myself, where that error is disabled (for my Windows machine I needed the custom client. I switched back to Linux a while ago and have the logs folder mounted as a nullfs now, which fixes the issue for me with minimal drawbacks, but I doubt that's possible using Windows). I found a somewhat confusing documentation on how to build the Nextcloud client, but I'm also not used to working with the tech-stack of the client, so not getting the docs might just be a "me" problem. ....Anyway, even so, I managed to find the one line that pops up that error message and yeets focus away from the active window every 5 minutes, and commented it out. That's all it took. Just had to search for the error message and I found it pretty quick (It was a while back, I dont remember the file or line number, and dont have the source checked out anymore to check.) Took me some hours actually getting it built, but I was able to, I imagine the Nextcloud team could get this done in about 5 minutes. If you can't push a real fix to this Issue for some years now, please consider just removing that error message so it becomes possible to work around this issue by manually denying writes to the log-folder. For all the linux users out there, I recommend mounting the log-folder as a filesystem that basically just deletes everything you write onto it... You wont get errors from Nextcloud, it doesn't even notice that something is wrong. I'm on Arch Linux, I installed this package from the AUR fuse-nullfs-git, and then put the following lines in my fstab: For different distros installation will ofc be different. |
|
the was a pull request #5780 addressing the logs issue and setting the useless "discovered" and "processing" logs to debug level.. for some reason major developer is reluctant to accept it. for now the only workaround remains to write the logs onto nullfs or ramdisk |
Bug description
First of all, thank you for developing this awesome and completely "free" home cloud solution! I use it on a daily basis.
Perhaps I'm alone with this, but my Nextcloud client generates >100 GB (800MB per sync run) worth of log files every day. I use an SSD; therefore this behaviour is seriously and unnecessarily (I never read these logs) reducing the lifespan of my hardware, as you surely know, SSD-s can be written to only a limited amount of times.
I am on Windows and using Virtual files support, so every file in the cloud gets a go for sync.
I have worked around this problem temporarily, by removing write permissions for every user for the logs folder, but now I get a pop-up Error from the client every half an hour or so, therefore I'm not satisfied with this solution. I had added the permissions back while writing this review, it generated 2.25 GB worth of logs in just 20 minutes. That's 162 GB of unnecessary data a day, contributing 57 TB a year, seriously bad for your SSD! That alone consumes an average SSD with 100TBW in less than 2 years!
Possible solution suggestions/ideas:
Cheap: I suggest adding an option in the client settings to disable the saving of log files on disk. And/Or, alternatively, add an option to save the logs in a compressed file format.
More time consuming, but better alternative: add an option for the verbosity of the logs, with the usual options for the loglevel: "debug" (the current behaviour), "only errors" - only logs unexpected errors,
The logs are VERY verbose, here is an excerpt:
I would expect the log to only contain information about special cases - being no smaller than a 100KB per sync run - not a detailed explanation of every atomic step during the sync process (this is a debug log, not a production log).
For the people stumbling across this issue, and are wondering: you can find the logs under
%appdata%\Nextcloud\logson Windows, orC:\Users\username\AppData\Roaming\Nextcloud\logs.Steps to reproduce
%appdata%\Nextcloud\logs. It should have the size of multiple Gigabytes.Expected behavior
I'd expect log files to only contain data about unexpected runtime errors that can help the team fix those errors. Not a detailed log of every atomic step during an otherwise uneventful sync.
Which files are affected by this bug
all of them
Operating system
Windows
Which version of the operating system you are running.
Windows 10 Pro 22H2, OS Build 19045.2364
Package
Appimage
Nextcloud Server version
25.0.2
Nextcloud Desktop Client version
3.4.1
Is this bug present after an update or on a fresh install?
Fresh desktop client install
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
Are you using an external user-backend?
Nextcloud Server logs
No response
Additional info
No response
The text was updated successfully, but these errors were encountered: