…rialization

**What this PR does / why we need it**:
This is a follow up to #11123 and
fixes a few bugs discovered by increasing the test coverage.

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [x] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

…cle (#11234)

**What this PR does / why we need it**:
This is a follow up from #11115
Instead of a creating a meta file per bloom creation, create a meta file
per compaction cycle.

**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

**What this PR does / why we need it**:
We want to control bloom compaction per tenant basis. Adding configs to
enable/disable bloom compactor.

**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

Fix nix configuration (mainly failing test), and break up various binaries (ie. logcli, promtail, loki, etc.) into their own pakcages.

---------

Co-authored-by: Trevor Whitney <[email protected]>

Only run the snyk pr comment workflow on PRs from branches, not on forks. We can't run this `on: pull_request_target` because in needs access to the `SNYK_TOKEN` secret, and when run `on: pull_request`, forks don't have permissions to comment on the PR (because they don't get the `GITHUB_TOKEN` secret.

**What this PR does / why we need it**:
Change per-pod latency panel unit from 'ms' to 's' (the metric is in the 's' unit)

…ipper is false (#11195)

**What this PR does / why we need it**:
When configuring tsdb shipper without using bolt db shipper, the ksonnet
library fails to generate the tsdb_shipper storage config.

Rebuilds tokenizers based on an iterable approach. This ends up being
significantly faster -- benchmarks below.

This PR doesn't remove the old tokenizers, just adds the new ones.
Revamping tokenizer consumers & removing the old variants can be done in
a separate PR.

Roughly, raw tokenizers complete in 66% of the time and chunkID prefixed
tokenizers in ~55-60% of the time.

```
pkg: github.com/grafana/loki/pkg/storage/bloom/v1
BenchmarkTokens/three/v1-10      	  134811	      8648 ns/op	       0 B/op	       0 allocs/op
BenchmarkTokens/three/v2-10      	  179797	      5728 ns/op	       0 B/op	       0 allocs/op
BenchmarkTokens/threeSkip1/v1-10 	  215822	      5260 ns/op	       0 B/op	       0 allocs/op
BenchmarkTokens/threeSkip1/v2-10 	  343784	      3514 ns/op	       0 B/op	       0 allocs/op
BenchmarkTokens/threeChunk/v1-10 	   82394	     14559 ns/op	       0 B/op	       0 allocs/op
BenchmarkTokens/threeChunk/v2-10 	  146973	      8138 ns/op	     128 B/op	       1 allocs/op
BenchmarkTokens/threeSkip1Chunk/v1-10      	  152475	      7878 ns/op	       0 B/op	       0 allocs/op
BenchmarkTokens/threeSkip1Chunk/v2-10      	  251373	      4776 ns/op	     128 B/op	       1 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/storage/bloom/v1	10.206s
```

returns the list of chunks in the bloom which failed the test so we can
merge these results across blocks

Adds a few utilities for comparing fingerprints to blocks that cover a
specific fingerprint range. Will likely need to be refactored with more
comprehensive types, but the logic still applies.

…9884)

**What this PR does / why we need it**:
Currently, we perform compaction and apply retention in the same loop.
Although we have a flag for not applying retention every time we perform
compaction, we still see compaction getting blocked when processing some
intensive delete requests(processed while applying retention).

This PR separates out the compaction and retention to run in a separate
loop. I have added a table-locking feature to avoid compaction and
retention from processing the same tables at a time. However, compaction
and retention would treat locked tables differently, as explained below:
* When compaction sees a table is locked: It would skip the table and
move on to the following table. However, before skipping, it would check
if the table has any uncompacted files and increment the newly added
counter called `loki_compactor_skipped_compacting_locked_tables_total`
to track how often we are skipping compacting tables which have
uncompacted files.
* When retention sees a table is locked: It would wait for the lock to
be released since we can't skip any tables while processing delete
requests.

**Special notes for your reviewer**:
* The check for tables with uncompacted files looks for count > 1
because initially, we did not support per tenant index in
`boltdb-shipper`, so a table can have a single compacted multi-tenant
index file. In a rare case where we have a single file which was
supposed to be compacted away, it is okay to have a single uncompacted
file for a while. The aim here is to not block compaction for too long
in a large cell with too many uncompacted files.
* Retention only works on the compacted index, so we first compact down
any uncompacted files while applying retention.

**Checklist**
- [x] Tests updated

---------

Co-authored-by: Ashwanth <[email protected]>

…1243)

**What this PR does / why we need it**:
logging: Add extra metadata to inflight requests

This adds extra metadata (similar to what we have in `metrics.go`) but
for queries in in-flight (both started and retrying)

Changes:
    Adds following data
    1. Query Hash
    2. Start and End time
    3. Start and End delta
    4. Length of the query
5. Moved the helper util to `queryutil` package because of cyclic
dependencies with `logql` package.
   
**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:
Find the screenshots of log entries looks like (both in `retry.go` and
`roundtrip.go`)

![Screenshot 2023-11-16 at 13 01
32](https://github.com/grafana/loki/assets/3735252/177e97ed-6ee8-41dd-b088-2e4f49562ba0)
![Screenshot 2023-11-16 at 13 02
15](https://github.com/grafana/loki/assets/3735252/fb328a37-dbe3-483e-b083-f21327858029)

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

---------

Signed-off-by: Kaviraj <[email protected]>

Co-authored-by: Periklis Tsirakidis <[email protected]>

Co-authored-by: Periklis Tsirakidis <[email protected]>

…11263)

**What this PR does / why we need it**:
In PR #9884, we separated the retention loop from compaction to avoid
blocking compaction for too long due to some intensive delete requests.
Currently, we track retention and compaction using the same metrics.
This PR adds separate metrics for monitoring retention operation. I have
also updated the Retention dashboard to use the new metrics.

The changes in #10688 did not
propage the trace ID from the context. `Frontend.RoundTripGRPC` would
inject the trace ID into the request. That's not done in `Frontend.Do`.
This changes extends the `codec.EncodeRequest` to inject the trace ID
there. This is more inline with other metadata.

**What this PR does / why we need it**:
Removes all usage of v1 tokenizers, renames v2 to v1 since we never
released this in a production way.

**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

**What this PR does / why we need it**:

In this PR, I fixed the grammatical mistake in an _index.md file.

**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

Signed-off-by: Bilal Khan <[email protected]>
Co-authored-by: J Stickler <[email protected]>

**What this PR does / why we need it**:

Fixing the small typo in the provided example
[here](https://grafana.com/docs/loki/latest/setup/install/helm/configure-storage/#configure-storage).
This PR just updates the doc.

**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

Co-authored-by: Michel Hollands <[email protected]>

…dex file names (#11277)

**What this PR does / why we need it**:
In PR #9884, we split the compaction and retention loop to run them
concurrently. Although we make sure we do not work on the same index
from compaction and retention loop, there is a chance that one could run
immediately after the other and finish quickly enough to build the index
with the same name as the previous one because in `boltdb-shipper`
index, we use epoch with `Seconds` precision while building the name of
the compacted index file. Since compaction uploads the new file first
and then deletes the old file, if the index is built with the same name,
we end up uploading the file and deleting it afterwards.

This PR fixes the issue by using ns precision for the timestamp in the
filenames.

**Special notes for your reviewer**:
This is not a problem for TSDB since we also add a checksum to the
filenames of the index during compaction.

…rialization