Make time format configruable.

[VihasMakwana]

• Issue: Logs pipeline operators logic assume UTC on kubernetes node, can cause incorrect timestamp in Splunk #644
• If the host is not set to UTC, the current logic breaks. To resolve this issue, the user needs to override current operators in agent.config section, and it wouldn't look so good.
• Allow users to modify the timestamp string in the filelog receiver. It uses current values by default, but give the option to override them through logsCollection.containers.timestampExtraction

[jvoravong]

Hey @VihasMakwana,
Can you describe in more detail what actually breaks or doesn't work as expected? Providing example logs and results would also be very helpful.

[VihasMakwana]

@jvoravong

Hey @VihasMakwana, Can you describe in more detail what actually breaks or doesn't work as expected? Providing example logs and results would also be very helpful.

From the linked issue:

• Our logs pipeline assumes kubernetes host will be in UTC. While this is definitely the most common scenario, and we do encourage users to always use UTC, if the host is not set to UTC, it breaks our hardcoded parsing logic.
• Ideally we should improve the regex to be able to handle the chance the host timestamp may not be UTC/Z and parse out the timestamp correctly.

Example of contained runtime with a different timezone:

2023-01-23T12:09:28.344779078-06:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"kube-system/microk8s.io-hostpath","level":"info","ts":"2023-01-23T18:09:28Z"}
2023-01-23T12:09:28.344779078-06:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"kube-system/microk8s.io-hostpath","level":"info","ts":"2023-01-23T18:09:28Z"}

The above wouldn't match the hardcoded regex.

For more details about reproducing this, refer #644 (comment).

[dmitryax]

Is the issue fixed now by #817 ? I would like to avoid introducing new configuration options if they are not strictly required

[jvoravong]

Left a similar comment as @dmitryax in the related issue. Was waiting for confirmation from involved reporters.

[VihasMakwana]

Closing this one, will raise another PR for #829