From 1e5107153c00416ab8543e7bc20679e1454cd5fd Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Thu, 4 Apr 2024 11:29:05 -0700 Subject: [PATCH] [receiver/filelog] Add documentation about contents of the offset tracking file (#31886) **Description:** This PR adds documentation explaining how to interpret the contents of the `filelog` receiver's offset tracking file, configured using the `storage` extension. **Link to tracking Issue:** #30970 --------- Co-authored-by: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com> Co-authored-by: Chris Mark Co-authored-by: Daniel Jaglowski --- receiver/filelogreceiver/README.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/receiver/filelogreceiver/README.md b/receiver/filelogreceiver/README.md index 4ee5d2820e51..b650098a982b 100644 --- a/receiver/filelogreceiver/README.md +++ b/receiver/filelogreceiver/README.md @@ -179,7 +179,17 @@ Exception in thread 2 "main" java.lang.NullPointerException ## Offset tracking -`storage` setting allows to define the proper storage extension to be used for storing file offsets. +The `storage` setting allows you to define the proper storage extension for storing file offsets. While the storage parameter can ensure that log files are consumed accurately, it is possible that logs are dropped while moving downstream through other components in the collector. For additional resiliency, see [Fault tolerant log collection example](../../examples/fault-tolerant-logs-collection/README.md) + +Here is some of the information the `filelog` receiver stores: +- The number of files it is currently tracking (`knownFiles`). +- For each file being tracked: + - The [fingerprint](../../pkg/stanza/fileconsumer/design.md#fingerprints) of the file (`Fingerprint.first_bytes`). + - The byte offset from the start of the file, indicating the position in the file from where the + `filelog` receiver continues reading the file (`Offset`). + - An arbitrary set of file attributes, such as the name of the file (`FileAttributes`). + +Exactly how this information is serialized depends on the type of storage being used. \ No newline at end of file