The logpipeline.telemetry.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to filter and ship application logs in Kyma. To get the current CRD and show the output in the YAML format, run this command:
kubectl get crd logpipeline.telemetry.kyma-project.io -o yamlThe following LogPipeline object defines a pipeline integrating with the HTTP/JSON-based output. It uses basic authentication and excludes application logs emitted by istio-proxy containers.
apiVersion: telemetry.kyma-project.io/v1alpha1
kind: LogPipeline
metadata:
name: custom-fluentd
generation: 2
spec:
input:
application:
containers:
exclude:
- istio-proxy
namespaces: {}
output:
http:
dedot: true
host:
valueFrom:
secretKeyRef:
key: Fluentd-endpoint
name: custom-fluentd
namespace: default
password:
valueFrom:
secretKeyRef:
key: Fluentd-password
name: custom-fluentd
namespace: default
tls: {}
uri: /customindex/kyma
user:
valueFrom:
secretKeyRef:
key: Fluentd-username
name: custom-fluentd
namespace: default
status:
conditions:
- lastTransitionTime: "2024-02-28T22:48:24Z"
message: Fluent Bit DaemonSet is ready
observedGeneration: 2
reason: AgentReady
status: "True"
type: AgentHealthy
- lastTransitionTime: "2024-02-28T22:48:11Z"
message: ""
observedGeneration: 2
reason: ConfigurationGenerated
status: "True"
type: ConfigurationGeneratedFor further examples, see the samples directory.
For details, see the LogPipeline specification file.
Spec:
| Parameter | Type | Description |
|---|---|---|
| files | []object | Provides file content to be consumed by a LogPipeline configuration |
| files.content | string | |
| files.name | string | |
| filters | []object | Describes a filtering option on the logs of the pipeline. |
| filters.custom | string | Custom filter definition in the Fluent Bit syntax. Note: If you use a custom filter, you put the LogPipeline in unsupported mode. |
| input | object | Defines where to collect logs, including selector mechanisms. |
| input.application | object | Configures in more detail from which containers application logs are enabled as input. |
| input.application.containers | object | Describes whether application logs from specific containers are selected. The options are mutually exclusive. |
| input.application.containers.exclude | []string | Specifies to exclude only the container logs with the specified container names. |
| input.application.containers.include | []string | Specifies to include only the container logs with the specified container names. |
| input.application.dropLabels | boolean | Defines whether to drop all Kubernetes labels. The default is false. |
| input.application.enabled | boolean | If enabled, application logs are collected. The default is true. |
| input.application.keepAnnotations | boolean | Defines whether to keep all Kubernetes annotations. The default is false. |
| input.application.keepOriginalBody | boolean | If the log attribute contains a JSON payload and it is successfully parsed, the log attribute will be retained if KeepOriginalBody is set to true. Otherwise, the log attribute will be removed from the log record. The default is true. |
| input.application.namespaces | object | Describes whether application logs from specific Namespaces are selected. The options are mutually exclusive. System Namespaces are excluded by default from the collection. |
| input.application.namespaces.exclude | []string | Exclude the container logs of the specified Namespace names. |
| input.application.namespaces.include | []string | Include only the container logs of the specified Namespace names. |
| input.application.namespaces.system | boolean | Set to true if collecting from all Namespaces must also include the system Namespaces like kube-system, istio-system, and kyma-system. |
| output | object | Fluent Bit output where you want to push the logs. Only one output can be specified. |
| output.custom | string | Defines a custom output in the Fluent Bit syntax. Note: If you use a custom output, you put the LogPipeline in unsupported mode. |
| output.http | object | Configures an HTTP-based output compatible with the Fluent Bit HTTP output plugin. |
| output.http.compress | string | Defines the compression algorithm to use. |
| output.http.dedot | boolean | Enables de-dotting of Kubernetes labels and annotations for compatibility with ElasticSearch based backends. Dots (.) will be replaced by underscores (_). Default is false. |
| output.http.format | string | Data format to be used in the HTTP request body. Default is json. |
| output.http.host | object | Defines the host of the HTTP receiver. |
| output.http.host.value | string | The value as plain text. |
| output.http.host.valueFrom | object | The value as a reference to a resource. |
| output.http.host.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| output.http.host.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| output.http.host.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| output.http.host.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
| output.http.password | object | Defines the basic auth password. |
| output.http.password.value | string | The value as plain text. |
| output.http.password.valueFrom | object | The value as a reference to a resource. |
| output.http.password.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| output.http.password.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| output.http.password.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| output.http.password.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
| output.http.port | string | Defines the port of the HTTP receiver. Default is 443. |
| output.http.tls | object | Configures TLS for the HTTP target server. |
| output.http.tls.ca | object | Defines an optional CA certificate for server certificate verification when using TLS. The certificate must be provided in PEM format. |
| output.http.tls.ca.value | string | The value as plain text. |
| output.http.tls.ca.valueFrom | object | The value as a reference to a resource. |
| output.http.tls.ca.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| output.http.tls.ca.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| output.http.tls.ca.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| output.http.tls.ca.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
| output.http.tls.cert | object | Defines a client certificate to use when using TLS. The certificate must be provided in PEM format. |
| output.http.tls.cert.value | string | The value as plain text. |
| output.http.tls.cert.valueFrom | object | The value as a reference to a resource. |
| output.http.tls.cert.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| output.http.tls.cert.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| output.http.tls.cert.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| output.http.tls.cert.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
| output.http.tls.disabled | boolean | Indicates if TLS is disabled or enabled. Default is false. |
| output.http.tls.key | object | Defines the client key to use when using TLS. The key must be provided in PEM format. |
| output.http.tls.key.value | string | The value as plain text. |
| output.http.tls.key.valueFrom | object | The value as a reference to a resource. |
| output.http.tls.key.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| output.http.tls.key.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| output.http.tls.key.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| output.http.tls.key.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
| output.http.tls.skipCertificateValidation | boolean | If true, the validation of certificates is skipped. Default is false. |
| output.http.uri | string | Defines the URI of the HTTP receiver. Default is "/". |
| output.http.user | object | Defines the basic auth user. |
| output.http.user.value | string | The value as plain text. |
| output.http.user.valueFrom | object | The value as a reference to a resource. |
| output.http.user.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| output.http.user.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| output.http.user.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| output.http.user.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
| variables | []object | A list of mappings from Kubernetes Secret keys to environment variables. Mapped keys are mounted as environment variables, so that they are available as Variables in the sections. |
| variables.name | string | Name of the variable to map. |
| variables.valueFrom | object | |
| variables.valueFrom.secretKeyRef | object | Refers to the value of a specific key in a Secret. You must provide name and namespace of the Secret, as well as the name of the key. |
| variables.valueFrom.secretKeyRef.key | string | The name of the attribute of the Secret holding the referenced value. |
| variables.valueFrom.secretKeyRef.name | string | The name of the Secret containing the referenced value |
| variables.valueFrom.secretKeyRef.namespace | string | The name of the Namespace containing the Secret with the referenced value. |
Status:
| Parameter | Type | Description |
|---|---|---|
| conditions | []object | An array of conditions describing the status of the pipeline. |
| conditions.lastTransitionTime (required) | string | lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. |
| conditions.message (required) | string | message is a human readable message indicating details about the transition. This may be an empty string. |
| conditions.observedGeneration | integer | observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. |
| conditions.reason (required) | string | reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. |
| conditions.status (required) | string | status of the condition, one of True, False, Unknown. |
| conditions.type (required) | string | type of condition in CamelCase or in foo.example.com/CamelCase. |
| unsupportedMode | boolean | Is active when the LogPipeline uses a custom output or filter; see unsupported mode. |
The status of the LogPipeline is determined by the condition types AgentHealthy, ConfigurationGenerated, and TelemetryFlowHealthy:
| Condition Type | Condition Status | Condition Reason | Condition Message |
|---|---|---|---|
| AgentHealthy | True | AgentReady | Fluent Bit agent DaemonSet is ready |
| AgentHealthy | True | RolloutInProgress | Pods are being started/updated |
| AgentHealthy | False | AgentNotReady | No Pods deployed |
| AgentHealthy | False | AgentNotReady | DaemonSet is not yet created |
| AgentHealthy | False | AgentNotReady | Failed to get DaemonSet |
| AgentHealthy | False | AgentNotReady | Pod is in the pending state because container: container name is not running due to: reason. Please check the container: container name logs. |
| AgentHealthy | False | AgentNotReady | Pod is in the failed state due to: reason |
| ConfigurationGenerated | True | AgentConfigured | LogPipeline specification is successfully applied to the configuration of Fluent Bit agent |
| ConfigurationGenerated | True | TLSCertificateAboutToExpire | TLS (CA) certificate is about to expire, configured certificate is valid until YYYY-MM-DD |
| ConfigurationGenerated | False | EndpointInvalid | HTTP output host invalid: reason |
| ConfigurationGenerated | False | ReferencedSecretMissing | One or more referenced Secrets are missing: Secret 'my-secret' of Namespace 'my-namespace' |
| ConfigurationGenerated | False | ReferencedSecretMissing | One or more keys in a referenced Secret are missing: Key 'my-key' in Secret 'my-secret' of Namespace 'my-namespace'" |
| ConfigurationGenerated | False | TLSCertificateExpired | TLS (CA) certificate expired on YYYY-MM-DD |
| ConfigurationGenerated | False | TLSConfigurationInvalid | TLS configuration invalid |
| ConfigurationGenerated | False | ValidationFailed | Pipeline validation failed due to an error from the Kubernetes API server |
| TelemetryFlowHealthy | True | FlowHealthy | No problems detected in the telemetry flow |
| TelemetryFlowHealthy | False | AllDataDropped | Backend is not reachable or rejecting logs. All logs are dropped. See troubleshooting: No Logs Arrive at the Backend |
| TelemetryFlowHealthy | False | BufferFillingUp | Buffer nearing capacity. Incoming log rate exceeds export rate. See troubleshooting: Agent Buffer Filling Up |
| TelemetryFlowHealthy | False | NoLogsDelivered | Backend is not reachable or rejecting logs. Logs are buffered and not yet dropped. See troubleshooting: No Logs Arrive at the Backend |
| TelemetryFlowHealthy | False | SomeDataDropped | Backend is reachable, but rejecting logs. Some logs are dropped. See troubleshooting: Not All Logs Arrive at the Backend |
| TelemetryFlowHealthy | False | ConfigurationNotGenerated | No logs delivered to backend because LogPipeline specification is not applied to the configuration of Fluent Bit agent. Check the 'ConfigurationGenerated' condition for more details |
| TelemetryFlowHealthy | Unknown | ProbingFailed | Could not determine the health of the telemetry flow because the self monitor probing failed |