0.4.0-rc.2: Follow-up error on start-up: lokiWriter failed to forward entry, channel was blocked

[jkroepke]

What's wrong?

I'm using import.http (new syntax in 0.40) together with logging and loki.write.

logging, import.http and loki.write are on the same local configuration.

If there is an error in a imported component, I getting an error from the logging block, too. This may confused people.

Steps to reproduce

1. setup config with import.file , logging and loki.write
2. have a syntax error or something else inside the imported module cause an could not perform the initial load successfully error

System information

Linux

Software version

Grafana Agent v0.40.0-rc.2

Configuration

Let me know, if I should provide one.

Logs

{"ts":"2024-02-28T09:36:39.878497528Z","level":"error","msg":"failed to evaluate config","controller_id":"all.config.this","trace_id":"7fe423b9ede149b939a284690834b4b3","node":"prometheus.remote_write.prometheus","err":"decoding River: all:107:17: unrecognized attribute name \"remote_username\""}
{"ts":"2024-02-28T09:36:39.878560328Z","level":"error","msg":"failed to evaluate config","controller_id":"","trace_id":"2b75bd4dd148b8c86d06d4c36fcadd83","node":"logging","err":"could not update logger: lokiWriter failed to forward entry, channel was blocked"}
Error: all:107:17: unrecognized attribute name "remote_username"

Error: /config/azure-server.river:48:1: Failed to evaluate node for config block: could not update logger: lokiWriter failed to forward entry, channel was blocked

47 |   
48 |   logging {
   |  _^^^^^^^^^
49 | |     level  = "warn"
50 | |     format = "json"
51 | |     write_to = [loki.write.remote.receiver]
52 | | }
   | |_^
53 |   loki.write "remote" {
interrupt received

Error: could not perform the initial load successfully

[JustNZ]

+1

[wildum]

I don't think this has something to do with modules, when I just run:

logging {
	write_to = [loki.write.remote.receiver]
}

loki.write "remote" {
    endpoint {
        url = "http://loki:3100/loki/api/v1/push"
    }
}

I'm getting the error "Error: ../config.river:1:1: Failed to evaluate node for config block: could not update logger: lokiWriter failed to forward entry, channel was blocked" and "Error: could not perform the initial load successfully"

We don't return directly on error, we continue evaluating so if you also have a broken module you will get the error for the module too.

[wildum]

now there seems to be a real bug with the logging config block introduced by #5992

[wildum]

I believe that I identified the problem: with #5992 we buffer the logs in the logger till it gets updated with the correct format. During the initial eval, it will already start buffering some lines of logs. When it gets updated, it will set the correct format/level and will empty its buffer. In this scenario, it needs to write the logs in the receiver of loki.write. For this we use an unbuffered channel. The problem is that loki.write is not ready to receive it because it's not running yet (we are still at the initial eval). => The logger fails to empty the buffered logs in the receiver channel, ending up with the error "lokiWriter failed to forward entry, channel was blocked"