Creation of empty directories #8
Comments
In fact, directory contents or disk-free-space are never checked, not before, not after writing of data. It takes the I think it is the simplest, fastest, and most predictable way - trying to focus on raw write speed. But I can definitely see these empty directories being a nuisance. At the moment I'd think that e.g. a |
|
I see. Thanks for explaining why the directories are created. I do see the point about speed and simplicity. A --prune option to vbs_rm seems like a reasonable suggestion I think. Perhaps that could then remove all empty directories on the vbs-disks? Otherwise, if I forget to prune, I will have to remember what experiments I have pruned and which not. And removing empty directories is usually harmless, especially when it is a special flag to the command - so not by accident. |
|
When digging into the codebase of the This is As can be seen the code was still under Please try if |
A few times I have found empty directories created by jive5ab on some disks on my flexbuff. One some disks I have the expected content like e.g.
$ ls /mnt/disk1/vo0202_oe_203-1615
vo0202_oe_203-1615.00000023 vo0202_oe_203-1615.00000052 vo0202_oe_203-1615.00000081
While on some other disks the corresponding folder e.g. "/mnt/disk2/vo0202_oe_203-1615" is empty. Having empty folders without files seem unnecessary to me. (And, although likely not a probable issue - it could in theory cause issues with number of inodes etc. I guess).
My hypothesis is that the empty folders turn up because of lack of disk-space. A "proper" flexbuff has a number of disks, which are only being used for recording. Preferably these disks are of equal size. This means that no disk will be full before all disks are full. However, for various reasons my flexbuff have disk with different amounts of free space. For example, disk1 can have 1TB free, and disk2 can have 65MB free. In these situations, jive5ab will - for each recording - check which disks are available and only write data to those disks with enough space. "enough" here is, I guess, the chunksize configured, which in my case is 256MB. In the example above, this would fit on disk1 but not on disk2. However, disk2 is not completely full, there is still space to create directories. Is it so that jive5ab creates an empty directory before checking if there is enough space to write a file into this directory? If so, this would explain the behaviour.
The interesting thing is that when I later run "vbs_rm -f" to remove data, it will not remove the empty directories. This is, perhaps, a separate issue. But I believe the empty directories should not be created in the first place.
It is easy-enough to get rid of them, just run "rmdir /mnt/disk*/*" since this will only remove empty directories. But still: they should not be created at all ;).
I cannot easily reproduce this issue, because as these are production machines data is created and removed on a regular basis. Currently, no disks are full enough to show this problem. But I still wanted to note this possible issue here, for future reference.
The text was updated successfully, but these errors were encountered: