Improve performance find zip archive

[johandahlberg]

This is an attempt to address #1662. Working on a code base that makes extensive us of zip archives containing parquet files that need to be read on the fly to load data, I noticed that find was a major bottleneck in that process.

I have made a zip specific implementation of find here that relies on the file list of the zip file, rather than explicitly walking.

Here is how I measured the performance:

from fsspec.implementations.zip import ZipFileSystem

# Example achieve with roughly 900 files, in some deep directory structures
file_system = ZipFileSystem("example.zip")
%timeit file_system.find("/")

Performance for find on current master

2.14 s ± 146 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Performance for find with this fix applied:

318 µs ± 11.5 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Let me know what you think.

[johandahlberg]

Thank you for the review @martindurant! I refactored the code quite a bit, to make use of the dir_cache instead of directly accessing the zip file. I think it made the code easier to follow.

Regarding removing the maxdepth that would be a change in the behavior compared to the find method on the AbstractFileSystem - since my initial strategy here was to try to implement something that had parity in the output I kept it in. If you still want me to remove it let me know and I'll do it (or feel free to push any changes you want to this branch if you prefer that).

I implemented it as suggested by you now, but just filtering for it at the end if it is set.

[martindurant]

I wonder, should this code go up into AbstractArchiveFileSystem, since it may have a similar performance benefit for libarchive or tar filesystems? I'm not sure if find is slow there or if there's anything else to worry about.

[johandahlberg]

@martindurant Let me know if you are happy with this or if there is anything else you'd like to see. When it comes to moving to evaluating this libarchive/tar file systems I unfortunately don't think that I will have the bandwidth to take that on.

[martindurant]

I unfortunately don't think that I will have the bandwidth to take that on.

Understood - maybe someone else gets the itch if indeed speed is a problem for the other ones.

[johandahlberg]

@martindurant thank you for a very enjoyable review process, and thanks from brining this in. It will help our use-case immensely.