c++: Chunk records are not immediately flushed to disk on write

[jhurliman]

Description

It's often repeated that chunk size should be dialed in for MCAP writing based on how much data you are willing to lose, traded off vs the performance overhead of writing data to disk too frequently, the decreased compression ratio, and the container+index overhead of additional Chunk records.

However, the current C++ implementation uses std::fwrite() which introduces an additional in-memory buffer (inside the standard library) before data is actually scheduled for committing to disk. This is fairly critical for unchunked mode where every 4 and 8 byte field of MCAP records turn into a std::fwrite() call, but it introduces a second unnecessary layer of buffering in chunked mode where the chunk is the in-memory buffer and std::fwrite() will also buffer and not necessarily flush to disk right away.

One option would be for the IWritable interface to expose a flush() method that is explicitly called after writing a chunk, which would in turn call std::fflush(file_) in the FileWriter implementation.

Note that some users are operating under the impression that unchunked mode will cause each message to immediately flush to disk, which is not the case and I don't see an easy fix to make this the case (if it's even desirable in the general case).

[linear]

FG-9267 c++: Chunk records are not immediately flushed to disk on write