Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional use of ISA-L for zlib decode/encode operations #1799

Open
belliriccardo opened this issue Jun 19, 2024 · 0 comments
Open

Optional use of ISA-L for zlib decode/encode operations #1799

belliriccardo opened this issue Jun 19, 2024 · 0 comments

Comments

@belliriccardo
Copy link

belliriccardo commented Jun 19, 2024

Is your feature request related to a problem? Please describe.

I have an application that requires the decoding of very large .blf files, with hundreds of millions of can messages. I've modified the BLFReaderclass a lot, but I have observed that a very large speedup can be achieved very easily just by using a more optimized zlib decompression library, such as Intel's ISA-L (and the bindings provided by the isal python package) as a drop-in replacement.

Describe the solution you'd like

The zlib import statement inside can/io/blf.py could be simply changed from:

import zlib

to:

try:
    from isal import isal_zlib as zlib
except ImportError:
    import zlib

This can be so that the library can be listed as optional in the installation instructions, and the remaining code can remain unmodified. I don't know about making it mandatory and/or replacing it altogether, as I haven't tested the compress routine (it isn't used in my specific application), but it should be fine.

Also I really haven't checked if the zlib module is used anywhere else, but obviously this change can be "universal".

Additional context

This speedup works best when you need "skip" a lot of messages in the log (e.g. if you're only interested in can messages with a specific arbitration id and skip all of the others), and therefore need to sift trough this .blf file very quickly. I don't have any precise figures on the % speedup because it is very much context dependent, but it can be significant, especially on such very large files; also the package's installed size is around 540KB, so not too heavy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant