Skip to content

Commit

Permalink
explain difference between flush and close
Browse files Browse the repository at this point in the history
Explain when flush is useful. Both has it purpose but the difference is
not obvious.

I find few places where flush is source of confusion:

ziglang/zig#13144
ziglang/zig#17863
  • Loading branch information
ianic committed Feb 12, 2024
1 parent 69caee2 commit 426011f
Show file tree
Hide file tree
Showing 2 changed files with 48 additions and 24 deletions.
1 change: 1 addition & 0 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Great materials for understanding deflate:
[RFC 1950 - zlib](https://datatracker.ietf.org/doc/html/rfc1950)
[RFC 1952 - gzip](https://datatracker.ietf.org/doc/html/rfc1952)
[zlib algorithm explained](https://github.com/madler/zlib/blob/643e17b7498d12ab8d15565662880579692f769d/doc/algorithm.txt)
[zlib manual](https://www.zlib.net/manual.html)
[Mark Adler on stackoverflow](https://stackoverflow.com/search?q=user%3A1180620+deflate)
[Faster zlib/DEFLATE](https://dougallj.wordpress.com/2022/08/20/faster-zlib-deflate-decompression-on-the-apple-m1-and-x86/)
[Reading bits with zero refill latency](https://dougallj.wordpress.com/2022/08/26/reading-bits-with-zero-refill-latency/)
Expand Down
71 changes: 47 additions & 24 deletions src/deflate.zig
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ fn Deflate(comptime container: Container, comptime WriterType: type, comptime Bl
prev_literal: ?u8 = null,

const Self = @This();

pub fn init(wrt: WriterType, level: Level) !Self {
const self = Self{
.wrt = wrt,
Expand Down Expand Up @@ -282,35 +283,23 @@ fn Deflate(comptime container: Container, comptime WriterType: type, comptime Bl
self.win.flush();
}

// Flush internal buffers to the output writer. Writes deflate block to
// the writer. Internal tokens buffer is empty after this.
pub fn flush(self: *Self) !void {
try self.tokenize(.flush);
}

// Slide win and if needed lookup tables.
fn slide(self: *Self) void {
const n = self.win.slide();
self.lookup.slide(n);
}

// Flush internal buffers and write deflate final block.
pub fn close(self: *Self) !void {
try self.tokenize(.final);
try container.writeFooter(&self.hasher, self.wrt);
}

pub fn setWriter(self: *Self, new_writer: WriterType) void {
self.block_writer.setWriter(new_writer);
self.wrt = new_writer;
}

// Writes all data from the input reader of uncompressed data.
// It is up to the caller to call flush or close if there is need to
// output compressed blocks.
/// Compresses as much data as possible, stops when the reader becomes
/// empty. It will introduce some output latency (reading input without
/// producing all output) because some data are still in internal
/// buffers.
///
/// It is up to the caller to call flush (if needed) or close (required)
/// when is need to output any pending data or complete stream.
///
pub fn compress(self: *Self, reader: anytype) !void {
while (true) {
// read from rdr into win
// Fill window from reader
const buf = self.win.writable();
if (buf.len == 0) {
try self.tokenize(.none);
Expand All @@ -320,19 +309,53 @@ fn Deflate(comptime container: Container, comptime WriterType: type, comptime Bl
const n = try reader.readAll(buf);
self.hasher.update(buf[0..n]);
self.win.written(n);
// process win
// Process window
try self.tokenize(.none);
// no more data in reader
// Exit when no more data in reader
if (n < buf.len) break;
}
}

/// Flushes internal buffers to the output writer. Outputs empty stored
/// block to sync bit stream to the byte boundary, so that the
/// decompressor can get all input data available so far.
///
/// It is useful mainly in compressed network protocols, to ensure that
/// deflate bit stream can be used as byte stream. May degrade
/// compression so it should be used only when necessary.
///
/// Completes the current deflate block and follows it with an empty
/// stored block that is three zero bits plus filler bits to the next
/// byte, followed by four bytes (00 00 ff ff).
///
pub fn flush(self: *Self) !void {
try self.tokenize(.flush);
}

/// Completes deflate bit stream by writing any pending data as deflate
/// final deflate block. HAS to be called once all data are written to
/// the compressor as a signal that next block has to have final bit
/// set.
///
pub fn close(self: *Self) !void {
try self.tokenize(.final);
try container.writeFooter(&self.hasher, self.wrt);
}

/// Use another writer while preserving history. Most probably flush
/// should be called on old writer before setting new.
pub fn setWriter(self: *Self, new_writer: WriterType) void {
self.block_writer.setWriter(new_writer);
self.wrt = new_writer;
}

// Writer interface

pub const Writer = io.Writer(*Self, Error, write);
pub const Error = BlockWriterType.Error;

// Write `input` of uncompressed data.
/// Write `input` of uncompressed data.
/// See compress.
pub fn write(self: *Self, input: []const u8) !usize {
var fbs = io.fixedBufferStream(input);
try self.compress(fbs.reader());
Expand Down

0 comments on commit 426011f

Please sign in to comment.