Provide fluent-bit-watcher container image that acts as a sidecar to any fluent-bit image

[reegnz]

Is your feature request related to a problem? Please describe.

Currently the fluent-operator project has to build a custom container image for fluent-bit whenever a new fluent-bit version is released. Sometimes this release is behind several weeks compared to the upstream.
Also when an organization already builds their own fluent-bit image with plugins they can't use their own fluent-bit image, they have to build another image with fluent-operator in mind as well.

Describe the solution you'd like

Instead of building off of the upstream fluent-bit container image, the fluent-bit-watcher should be a standalone container image.
It should not act as a process supervisor (which it does today). It should only concern itself with notifying fluent-bit to reload it's config. This new image could be run as a sidecar to the upstream fluent-bit image, or any customized fluent-bit image.

The sidecar and the main container can have the same configuration mounted. If the configuration changes, the sidecar triggers a hot-reload by either:

• POST request to the main container process eg. 127.0.0.1:2020/api/v1/reload.
• sending a SIGHUP to the fluent-bit process (using shareProcessNamespace: true in the pod spec)

Describe alternatives you've considered
Building my own 'reloader' sidecar of my own so I can use a different fluent-bit image instead of the downstream rebuild published by fluent-operator.

Additional context
By not re-packaging fluent-bit, the release pressure of having to keep pushing releases of that image is completely eliminated. Images built by fluent-operator are fully decoupled from the patch release cadence of fluent-bit.

I'm not sure if the same approach could be done with fluentd using HTTP POST, but the shared process namespace approach is still doable.

[reegnz]

Ah actually I wanted to open this in fluent-operator: fluent/fluent-operator#1373