Provide fluent-bit-watcher container image that acts as a sidecar to any fluent-bit image

[reegnz]

Is your feature request related to a problem? Please describe.

Currently the fluent-operator project has to build a custom container image for fluent-bit whenever a new fluent-bit version is released. Sometimes this release is behind several weeks compared to the upstream.
Also when an organization already builds their own fluent-bit image with plugins they can't use their own fluent-bit image, they have to build another image with fluent-operator in mind as well.

Describe the solution you'd like

Instead of building off of the upstream fluent-bit container image, the fluent-bit-watcher should be a standalone container image.
It should not act as a process supervisor (which it does today). It should only concern itself with notifying fluent-bit to reload it's config. This new image could be run as a sidecar to the upstream fluent-bit image, or any customized fluent-bit image.

The sidecar and the main container can have the same configuration mounted. If the configuration changes, the sidecar triggers a hot-reload by either:

• POST request to the main container process eg. 127.0.0.1:2020/api/v1/reload.
• sending a SIGHUP to the fluent-bit process (using shareProcessNamespace: true in the pod spec)

Additional context

I've considered building my own 'reloader' sidecar of my own so I can use a different fluent-bit image instead of the downstream rebuild published by fluent-operator.

By not re-packaging fluent-bit, the release pressure of having to keep pushing releases of that image is completely eliminated. Images built by fluent-operator are fully decoupled from the patch release cadence of fluent-bit.

I'm not sure if the same approach could be done with fluentd using HTTP POST, but the shared process namespace approach is still doable.

[benjaminhuo]

@wanjunlei @Gentleelephant @wenchajun @joshuabaird what do you think?

[joshuabaird]

I think this is a good idea. It's not ideal that fluent-operator has to roll it's own fluent-bit and fluent images for the reasons that @reegnz stated. The shared process namespace approach may be better here simply because we can keep the reload process the same for both fluent-bit and fluentd.

[reegnz]

Actually, I think at this point we can probably just adopt https://github.com/jimmidyson/configmap-reload and be done with it. Fluent-bit can listen on localhost for reload requests, the jimmydyson image can deal with the watch and notify logic.

The jimmydyson image does only support HTTP messages currently, but I'd use that for fluent-bit. That leaves fluentd that still needs SIGHUP. Although looking at that project, they write that OS SIGNAL support is expected feature in the future.

I'm going to experiment with this idea a bit.

[reegnz]

See github issue for adding OS Signal support to the jimmydyson/configmap-reload program: jimmidyson/configmap-reload#23