NSFS | Regular object key & dir content object - key/ co-existence and overriding

[romayalon]

Environment info

• NooBaa Version: Master
• Platform: NC

key and key/ co-existence -

Actual behavior

In general - On AWS key and key/ are 2 different objects, we can have both simultaneously while in NooBaa we didn't handle this edge case due to file system limitation, when trying to create obj1 and obj1/ directly on the FS we will get the following error -

% echo blbkabkab > obj1
% mkdir obj1/
mkdir: .: Not a directory

Expected behavior

If we do want to support it, it'll require a special design.

Key and Key/ overriding bugs -

Actual behavior

There are 2 faulty scenarios that were found, related to overriding of key and key/ -

1. PUT obj1.txt and then PUT obj1.txt/ - throws an error - NoSuchObject.
2. PUT obj2.txt/ and then PUT obj2.txt - throws an error - InternalError.

Steps to reproduce -

1. Tab 1 - Deploy dev NooBaa NC - sudo node src/cmd/nsfs --debug=5
2. Tab 2 -
   1. Create an account using NooBaa CLI - sudo node src/cmd/manage_nsfs.js account add --name account1 --user root --new_buckets_path=/private/tmp/dir1/ --debug 5
   2. Create a bucket using NooBaa CLI - sudo node src/cmd/manage_nsfs.js bucket add --name bucket1 --owner account1 --path /private/tmp/dir1/
   3. init AWS S3api alias pointing to NooBaa having access keys of account1 - alias s3api='AWS_ACCESS_KEY_ID=<access_key> AWS_SECRET_ACCESS_KEY=<secret_key> aws --endpoint-url=https://localhost:6443 --no-verify-ssl s3api'
   4. From here there are 2 scenarios -
      1. PUT obj1.txt and then PUT obj1.txt/ -
         1. PUT obj1.txt - s3api put-object --bucket=bucket1 --key=obj1.txt --body=obj1.txt
         2. Override obj1.txt with obj1.txt/ - s3api put-object --bucket=bucket1 --key=obj1.txt/ --body=obj1.txt
         3. NooBaa throws an error - NoSuchObject - in the logs it looks like we got ENOTDIR when renaming the new object file to obj1.txt/.folder.
            [Error: Not a directory] { code: 'ENOTDIR', context: 'Rename _old_path=/private/tmp/dir2/.noobaa-nsfs_66d04430da7de6f67dc274ec/uploads/9b718dce-05b7-49a8-ad06-73cafbf78ad9 _new_path=/private/tmp/dir2/obj1.txt/.folder ' }
      2. PUT obj2.txt/ and then PUT obj2.txt -
         1. PUT ob21.txt/ - s3api put-object --bucket=bucket1 --key=obj2.txt/ --body=obj2.txt
         2. Override obj1.txt with obj1.txt/ - s3api put-object --bucket=bucket1 --key=obj2.txt --body=obj2.txt
         3. NooBaa throws an error - InternalError - in the logs it looks like we got EISDIR when renaming the new object file to obj1.txt/.folder.
            [Error: Is a directory] { code: 'EISDIR', context: 'Rename _old_path=/private/tmp/dir1/.noobaa-nsfs_66d04430da7de6f67dc274ec/uploads/0aafbda8-6ad0-40d3-815f-24abddacacb1 _new_path=/private/tmp/dir1/obj2.txt ' }

Expected behavior

1. Handle these scenarios correctly.

[hseipp]

@romayalon Great that you reported that issue - I was also about to report this when I had to change Ceph s3-tests test_bucket_list*_delimiter*() functions code to make it work with NSFS. Apart from AWS, Ceph and IBM Cloud (COS) also allow keys and prefixes with the same name, which obviously does not directly map to POSIX file systems as there a directory entry can not be a file and directory at the same time.

[romayalon]

Thanks @hseipp for the great findings!
we need to come up with an indirect FS mapping for the key and key/ co-existence fix, but we can definitely fix the "overrides" scenarios with the existing code. Did you find it as part of the versioning tests effort?

CC: @guymguym

[hseipp]

Yes, as part of my research for the versioning tests I ran the complete https://github.com/ceph/s3-tests against NC and inspected the results. This is where I came across the above issue.

[romayalon]

@nadavMiz I remember vaguely that we didn't support directory object and versioning together, can you please check that?

[romayalon]

@naveenpaul1 will take the overriding issues