You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I have an application that requires the decoding of very large .blf files, with hundreds of millions of can messages. I've modified the BLFReaderclass a lot, but I have observed that a very large speedup can be achieved very easily just by using a more optimized zlib decompression library, such as Intel's ISA-L (and the bindings provided by the isal python package) as a drop-in replacement.
Describe the solution you'd like
The zlib import statement inside can/io/blf.py could be simply changed from:
import zlib
to:
try:
from isal import isal_zlib as zlib
except ImportError:
import zlib
This can be so that the library can be listed as optional in the installation instructions, and the remaining code can remain unmodified. I don't know about making it mandatory and/or replacing it altogether, as I haven't tested the compress routine (it isn't used in my specific application), but it should be fine.
Also I really haven't checked if the zlib module is used anywhere else, but obviously this change can be "universal".
Additional context
This speedup works best when you need "skip" a lot of messages in the log (e.g. if you're only interested in can messages with a specific arbitration id and skip all of the others), and therefore need to sift trough this .blf file very quickly. I don't have any precise figures on the % speedup because it is very much context dependent, but it can be significant, especially on such very large files; also the package's installed size is around 540KB, so not too heavy.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
I have an application that requires the decoding of very large .blf files, with hundreds of millions of can messages. I've modified the
BLFReader
class a lot, but I have observed that a very large speedup can be achieved very easily just by using a more optimized zlib decompression library, such as Intel's ISA-L (and the bindings provided by theisal
python package) as a drop-in replacement.Describe the solution you'd like
The zlib import statement inside
can/io/blf.py
could be simply changed from:to:
This can be so that the library can be listed as optional in the installation instructions, and the remaining code can remain unmodified. I don't know about making it mandatory and/or replacing it altogether, as I haven't tested the
compress
routine (it isn't used in my specific application), but it should be fine.Also I really haven't checked if the
zlib
module is used anywhere else, but obviously this change can be "universal".Additional context
This speedup works best when you need "skip" a lot of messages in the log (e.g. if you're only interested in can messages with a specific arbitration id and skip all of the others), and therefore need to sift trough this .blf file very quickly. I don't have any precise figures on the % speedup because it is very much context dependent, but it can be significant, especially on such very large files; also the package's installed size is around 540KB, so not too heavy.
The text was updated successfully, but these errors were encountered: