fix(operator): Fix structured logs common fields

[periklis]

What this PR does / why we need it:
The Loki Operator is using the logr.Logger interface extensively to emit structured logs per controller and handler. In addition the controller-runtime is leveraging the Builder factory to link reconciliation errors (i.e. any kind of error that is bubbled up and returned by the controllers' Reconcile() method) to the managing controller. However the present implementation makes it hard to link top-bottom logs for a specific combo of a controller, a handler and a LokiStack name due to the following issues:

• The controller-runtime constructs a custom logger with fields extracted from Builder.For() and Builder.Named methods. This approach works when linking reconciler errors to the orginating controller/LokiStack if and only if the controller is built with Builder.For(lokiv1.LokiStack{} as the source type. However this is not the case for controllers like the AlertingRuleReconciler, RecordingRuleReconciler and more critical for the CertRotationReconciler rendering searching for errors per LokiStack impossible.
• Although a logger sink per controller is provided in main.go the resulting logger instance (used for info/warning logging/non-reconciliation-error logging) needs to be manually augmented with similar fields (controller, lokistack.name, lokistack.namespace) per controller, handler and incoming request. This manual augmentation happens via logr.Logger.WithValues() and has been not consistent.

With the following PR the implementation is streamlined to:

1. Use the same set of fields to identify the controller, the handler (called event), the lokistack (name and namespace) whether the structured log emitted is a reconciler error from controller-runtime or an info/warning log from the Loki Operator code base.
2. Controllers without a Builder.For(lokiv1.LokiStack{}) setup will use consistently the same fields for lokistack (name and namespace) wherever applicable, i.e. AlertingRuleReconciler, RecordingRuleReconciler and RulerConfigReconciler are manually mimicking the same fields to link annotation error with the target LokiStack.

Examples (for the sake of simplicity modeled with jq):

• Searching for a lokistack's events when rotate certificates:
  jq -c 'select(.lokistack.name=="lokistack-dev" and .event=="createOrRotateCertificates")'
• Searching for all controller logs (info/error) for a handler:
  jq -c 'select(.controller=="lokistack" and .event=="createOrUpdateLokiStack")'
• Search for all reconciler errors for a single lokistack for a controller:
  jq -c 'select(.controller=="certrotation" and ._message=="Reconciler error" and .lokistack.name=="lokistack-dev")'

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
This PR leaves out the duplicates cleanup of usage references of kverrors.Wrap, log.Info and log.Error across the code base where the Lokistack identifiers (name, namespace) might be logger under an additional key (e.g. sometimes key, ref, name, etc.). This is left as future work in a separate PR. ✔️

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
  • If the change is worth mentioning in the release notes, add add-to-release-notes label
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[JoaoBraveCoding]

Food for thought, IIUC our logs will benefit from these event values, but if an error occurs in a handler they will not have event values. Should we add event values when we throw an error?

[JoaoBraveCoding]

lgtm

[xperimental]

We usually use prefixes to group similar constants together. Does this make sense here as well? (calling them ControllerNameAlertingRule, etc., for example)

[xperimental]

Small typo and suggestion for renaming (don't know if we did a similar thing already):

Suggested change

	type LogContructorType func(request *reconcile.Request) logr.Logger
	type LogConstructorFunc func(request *reconcile.Request) logr.Logger

---

Also this type does not need to be public, as far as I can see...

[xperimental]

I wonder why we need this, when we have the lokiStackLogConstructor above which should already produce a lokistack field. Do we maybe just need to get our logger from the Context (there's a helper ctrl.LoggerFrom(ctx))?

[xperimental]

I wonder if we can avoid the extra dependency to klog just for getting an ObjectRef to produce a string representation of namespace/name.

--edit: Oh, wait it's not producing namespace/name but an object with namespace and name attributes in it. Is that what we want?

[xperimental]

I recently wondered if we should remove a few of these outputs that happen every reconciliation and only print output if there's anything "noteworthy". Don't know if this should also happen in this PR though.

[periklis]

Closing for a later stage to review this