GH-37555: [Python] Update get_file_info_selector to ignore base direc…

…tory (#37558) ### Rationale for this change There has been some changes in the way fsspec lists the directories with new version 2023.9.0, see fsspec/filesystem_spec#1329, which caused our tests to start failing. ### What changes are included in this PR? This PR updates the `get_file_info_selector` in [FSSpecHandler](https://arrow.apache.org/docs/_modules/pyarrow/fs.html#FSSpecHandler) class to keep the behaviour of our spec. ### Are there any user-facing changes? No. * Closes: #37555 Authored-by: AlenkaF <[email protected]> Signed-off-by: AlenkaF <[email protected]>
apache · Sep 14, 2023 · 396b475 · 396b475
1 parent 15a8ac3
commit 396b475
Showing 1 changed file with 6 additions and 1 deletion.
diff --git a/python/pyarrow/fs.py b/python/pyarrow/fs.py
@@ -356,7 +356,12 @@ def get_file_info_selector(self, selector):
             selector.base_dir, maxdepth=maxdepth, withdirs=True, detail=True
         )
         for path, info in selected_files.items():
-            infos.append(self._create_file_info(path, info))
+            _path = path.strip("/")
+            base_dir = selector.base_dir.strip("/")
+            # Need to exclude base directory from selected files if present
+            # (fsspec filesystems, see GH-37555)
+            if _path != base_dir:
+                infos.append(self._create_file_info(path, info))
 
         return infos