Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mkdir does not create dirs with herarchical namespace enabled #203

Closed
lmmentel opened this issue Mar 28, 2021 · 5 comments
Closed

mkdir does not create dirs with herarchical namespace enabled #203

lmmentel opened this issue Mar 28, 2021 · 5 comments

Comments

@lmmentel
Copy link

lmmentel commented Mar 28, 2021

I have a storage account (V2) with hierarchical namespace enabled and cannot get any of the mkdir, mkdirs, makedir or makedirs to actually create a directory:

Here is a minimal example:

fs = fsspec.filesystem('abfs', connection_string=connection_string)
fs.makedir("test/test-dir/")

What happened:

In both cases, when test container exists and if it does not, behavior is the same and without exist_ok=True gives me

...
StorageErrorException: Operation returned an invalid status 'Server encountered an internal error. Please try again after some time.'
...
During handling of the above exception, another exception occurred:

FileExistsError: Cannot overwrite existing Azure container -- test already exists.                         with Azure error Server encountered an internal error. Please try again after some time.
RequestId:efdce538-d01e-0002-0f21-24942d000000
Time:2021-03-28T22:23:39.9321554Z
ErrorCode:InternalError
Error:None

With exist_ok=True it just hangs for some time, does not give any output and does not create the directory.

What you expected to happen:

To the directory to be created for the specified path.

Anything else we need to know?:

I went through #137 and can create directories when hierarchical namespace option is disabled but then even when I include a trailing / a bunch of empty <no_name> files are produced in the dirs. I guess this is an artifact of creating virtual dirs.

Environment:

  • Dask version: 2021.3.1
  • Python version: 3.8.5
  • Operating System: linux mint 20.1
  • Install method (conda, pip, source): poetry
@hayesgb
Copy link
Collaborator

hayesgb commented Mar 31, 2021

Based on this discussion on fsspec, release 0.7 of adlfs will make mkdir a noop, to align be consistent with fsspec, and more closely align to the Azure blob API.

@lmmentel
Copy link
Author

lmmentel commented Apr 4, 2021

Thanks, a note somewhere in the docs would be nice, I guess since this is one of the few issues mentioning mkdir.

Could you share your thoughts on the different behavior with hierarchical namespace enabled and not?

@hayesgb
Copy link
Collaborator

hayesgb commented Apr 9, 2021

Based on my experience, I would not expect there to be a difference. I our environment, we have hierarchical namespaces enabled and the functionality works. The only time we've had a issue is if permissions do not allow the ServicePrincipal access to a given container / folder / directory.

@hayesgb hayesgb closed this as completed May 7, 2021
@hayesgb hayesgb reopened this May 7, 2021
@hayesgb
Copy link
Collaborator

hayesgb commented May 7, 2021

Closing this as it seems to be stale.

@hayesgb hayesgb closed this as completed May 7, 2021
@hfurkanvural
Copy link
Contributor

hfurkanvural commented Feb 2, 2023

It seems the issue persists. I m trying to upload a directory recursively and the function call on this line hangs forever when it is a directory. On the storage account, hierarchical namespace is disabled and I connect to azure storage by using access keys (so I assume there is no problem regarding the permission).

I see this PR changes this function call to return and it works, but the rest of the changes are obsolete with the latest release.
It seems there is an issue with async calls but unfortunately I dont know what is it exactly. As the function is inherited from fsspec and it is a noop function, I expect it to return immediately but that is not the case.

Environment:
Version: 2023.1.0
Python version: 3.8.10
Operating System: ubuntu20.04
Install method (conda, pip, source): poetry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants