From aafb77a82f53bc1a4371964a41d70b3738c17383 Mon Sep 17 00:00:00 2001 From: jack-berg <34418638+jack-berg@users.noreply.github.com> Date: Fri, 5 Apr 2024 10:21:15 -0500 Subject: [PATCH] Define OTEL_EXPERIMENTAL_CONFIG_FILE to ignore other env vars, add env var substitution default syntax (#3948) Fixes #3752. Implementation of the configuration working group recommendation as [described here](https://github.com/open-telemetry/opentelemetry-specification/issues/3752#issuecomment-1995582317). - Adds definition for `OTEL_EXPERIMENTAL_CONFIG_FILE `, which ignores other env vars when evaluating, except env var substitution references - Adds new env var substitution default syntax `${ENVVAR:-defaultValue}` - Paired with https://github.com/open-telemetry/opentelemetry-configuration/pull/76, which defines a new [sdk-config.yaml](https://github.com/jack-berg/opentelemetry-configuration/blob/starter-template/examples/sdk-config.yaml) starter template referencing all env vars that map cleanly. --- spec-compliance-matrix.md | 63 +++++------ .../configuration/file-configuration.md | 11 +- .../sdk-environment-variables.md | 104 +++++++++++++----- 3 files changed, 116 insertions(+), 62 deletions(-) diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index 94bd625ceb1..733c607371e 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -269,37 +269,38 @@ Disclaimer: this list of features is still a work in progress, please refer to t Note: Support for environment variables is optional. -| Feature | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift | -|----------------------------------------------------------|-----|------|-----|-------------|------|--------|-----|------|-----|------|-------| -| OTEL_SDK_DISABLED | - | + | - | - | - | - | + | - | - | - | - | -| OTEL_RESOURCE_ATTRIBUTES | + | + | + | + | + | + | + | + | + | + | - | -| OTEL_SERVICE_NAME | + | + | + | + | + | + | + | | + | + | | -| OTEL_LOG_LEVEL | - | - | + | [-][py1059] | + | - | + | | - | - | - | -| OTEL_PROPAGATORS | - | + | | + | + | + | + | - | - | - | - | -| OTEL_BSP_* | + | + | + | + | + | + | + | + | - | + | - | -| OTEL_BLRP_* | | + | | | | | | + | | + | | -| OTEL_EXPORTER_OTLP_* | + | + | | + | + | + | + | + | + | + | - | -| OTEL_EXPORTER_ZIPKIN_* | - | + | | + | + | - | + | - | - | + | - | -| OTEL_TRACES_EXPORTER | - | + | + | + | + | + | + | - | - | - | | -| OTEL_METRICS_EXPORTER | - | + | | + | - | - | + | - | - | - | - | -| OTEL_LOGS_EXPORTER | - | + | | + | | | + | | | - | | -| OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT | + | + | + | + | + | + | + | + | - | + | | -| OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT | + | + | + | + | + | + | + | | | + | | -| OTEL_SPAN_EVENT_COUNT_LIMIT | + | + | + | + | + | + | + | + | - | + | | -| OTEL_SPAN_LINK_COUNT_LIMIT | + | + | + | + | + | + | + | + | - | + | | -| OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT | + | - | | + | + | + | + | | | + | | -| OTEL_LINK_ATTRIBUTE_COUNT_LIMIT | + | - | | + | + | + | + | | | + | | -| OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT | | | | | | | + | | | | | -| OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT | | | | | | | + | | | | | -| OTEL_TRACES_SAMPLER | + | + | + | + | + | + | + | - | - | - | | -| OTEL_TRACES_SAMPLER_ARG | + | + | + | + | + | + | + | - | - | - | | -| OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT | + | + | + | + | + | - | + | | | + | | -| OTEL_ATTRIBUTE_COUNT_LIMIT | + | + | + | + | + | - | + | | | + | | -| OTEL_METRIC_EXPORT_INTERVAL | - | + | | | | | + | | | + | | -| OTEL_METRIC_EXPORT_TIMEOUT | - | - | | | | | + | | | + | | -| OTEL_METRICS_EXEMPLAR_FILTER | - | + | | | | | + | | | - | | -| OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE | - | + | + | + | | | + | | | + | | -| OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION | | + | | | | | | | | | | +| Feature | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift | +|----------------------------------------------------------|----|------|----|-------------|------|--------|-----|------|-----|------|-------| +| OTEL_SDK_DISABLED | - | + | - | - | - | - | + | - | - | - | - | +| OTEL_RESOURCE_ATTRIBUTES | + | + | + | + | + | + | + | + | + | + | - | +| OTEL_SERVICE_NAME | + | + | + | + | + | + | + | | + | + | | +| OTEL_LOG_LEVEL | - | - | + | [-][py1059] | + | - | + | | - | - | - | +| OTEL_PROPAGATORS | - | + | | + | + | + | + | - | - | - | - | +| OTEL_BSP_* | + | + | + | + | + | + | + | + | - | + | - | +| OTEL_BLRP_* | | + | | | | | | + | | + | | +| OTEL_EXPORTER_OTLP_* | + | + | | + | + | + | + | + | + | + | - | +| OTEL_EXPORTER_ZIPKIN_* | - | + | | + | + | - | + | - | - | + | - | +| OTEL_TRACES_EXPORTER | - | + | + | + | + | + | + | - | - | - | | +| OTEL_METRICS_EXPORTER | - | + | | + | - | - | + | - | - | - | - | +| OTEL_LOGS_EXPORTER | - | + | | + | | | + | | | - | | +| OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT | + | + | + | + | + | + | + | + | - | + | | +| OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT | + | + | + | + | + | + | + | | | + | | +| OTEL_SPAN_EVENT_COUNT_LIMIT | + | + | + | + | + | + | + | + | - | + | | +| OTEL_SPAN_LINK_COUNT_LIMIT | + | + | + | + | + | + | + | + | - | + | | +| OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT | + | - | | + | + | + | + | | | + | | +| OTEL_LINK_ATTRIBUTE_COUNT_LIMIT | + | - | | + | + | + | + | | | + | | +| OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT | | | | | | | + | | | | | +| OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT | | | | | | | + | | | | | +| OTEL_TRACES_SAMPLER | + | + | + | + | + | + | + | - | - | - | | +| OTEL_TRACES_SAMPLER_ARG | + | + | + | + | + | + | + | - | - | - | | +| OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT | + | + | + | + | + | - | + | | | + | | +| OTEL_ATTRIBUTE_COUNT_LIMIT | + | + | + | + | + | - | + | | | + | | +| OTEL_METRIC_EXPORT_INTERVAL | - | + | | | | | + | | | + | | +| OTEL_METRIC_EXPORT_TIMEOUT | - | - | | | | | + | | | + | | +| OTEL_METRICS_EXEMPLAR_FILTER | - | + | | | | | + | | | - | | +| OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE | - | + | + | + | | | + | | | + | | +| OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION | | + | | | | | | | | | | +| OTEL_EXPERIMENTAL_CONFIG_FILE | | | | | | | | | | | | ## File Configuration diff --git a/specification/configuration/file-configuration.md b/specification/configuration/file-configuration.md index 6e75166befc..4b34f951ed0 100644 --- a/specification/configuration/file-configuration.md +++ b/specification/configuration/file-configuration.md @@ -72,7 +72,7 @@ Configuration files support environment variables substitution for references which match the following regular expression: ```regexp -\$\{(?[a-zA-Z_][a-zA-Z0-9_]*)} +\$\{(?[a-zA-Z_][a-zA-Z0-9_]*)(:-(?[^\n]*))?} ``` The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed @@ -84,8 +84,11 @@ invalid. Environment variable substitution MUST only apply to scalar values. Mapping keys are not candidates for substitution. -If a referenced environment variable is not defined, it MUST be replaced with an -empty value. +The `DEFAULT_VALUE` is an optional fallback value which is substituted +if `ENV_NAME` is null, empty, or undefined. `DEFAULT_VALUE` consists of 0 or +more non line break characters (i.e. any character except `\n`). If a referenced +environment variable is not defined and does not have a `DEFAULT_VALUE`, it MUST +be replaced with an empty value. Node types MUST be interpreted after environment variable substitution takes place. This ensures the environment string representation of boolean, integer, @@ -119,6 +122,7 @@ bool_key: ${BOOl_VALUE} # Valid reference to BOOl_ int_key: ${INT_VALUE} # Valid reference to INT_VALUE float_key: ${FLOAT_VALUE} # Valid reference to FLOAT_VALUE combo_string_key: foo ${STRING_VALUE} ${FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE +string_key_with_default: ${UNDEFINED_KEY:-fallback} # UNDEFINED_KEY is not defined but a default value is included undefined_key: ${UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with "" ${STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored recursive_key: ${REPLACE_ME} # Valid reference to REPLACE_ME @@ -135,6 +139,7 @@ bool_key: true # Interpreted as type bool, tag URI int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str +string_key_with_default: fallback # Interpreted as type string, tag URI tag:yaml.org,2002:str undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null ${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str recursive_key: ${DO_NOT_REPLACE_ME} # Interpreted as type string, tag URI tag:yaml.org,2002:str diff --git a/specification/configuration/sdk-environment-variables.md b/specification/configuration/sdk-environment-variables.md index ef4794cb046..30c3392c551 100644 --- a/specification/configuration/sdk-environment-variables.md +++ b/specification/configuration/sdk-environment-variables.md @@ -8,7 +8,39 @@ aliases: # OpenTelemetry Environment Variable Specification -**Status**: [Mixed](../document-status.md) +**Status**: [Stable](../document-status.md) except where otherwise specified + +
+Table of Contents + + + +- [Implementation guidelines](#implementation-guidelines) +- [Parsing empty value](#parsing-empty-value) +- [Special configuration types](#special-configuration-types) + * [Boolean value](#boolean-value) + * [Numeric value](#numeric-value) + * [Enum value](#enum-value) + * [Duration](#duration) +- [General SDK Configuration](#general-sdk-configuration) +- [Batch Span Processor](#batch-span-processor) +- [Batch LogRecord Processor](#batch-logrecord-processor) +- [Attribute Limits](#attribute-limits) +- [Span Limits](#span-limits) +- [LogRecord Limits](#logrecord-limits) +- [OTLP Exporter](#otlp-exporter) +- [Zipkin Exporter](#zipkin-exporter) +- [Prometheus Exporter](#prometheus-exporter) +- [Exporter Selection](#exporter-selection) +- [Metrics SDK Configuration](#metrics-sdk-configuration) + * [Exemplar](#exemplar) + * [Periodic exporting MetricReader](#periodic-exporting-metricreader) +- [File Configuration](#file-configuration) +- [Language Specific Environment Variables](#language-specific-environment-variables) + + + +
The goal of this specification is to unify the environment variable names between different OpenTelemetry implementations. @@ -17,22 +49,16 @@ If they do, they SHOULD use the names listed in this document. ## Implementation guidelines -**Status**: [Stable](../document-status.md) - Environment variables MAY be handled (implemented) directly by a component, in the SDK, or in a separate component (e.g. environment-based autoconfiguration component). The environment-based configuration MUST have a direct code configuration equivalent. ## Parsing empty value -**Status**: [Stable](../document-status.md) - The SDK MUST interpret an empty value of an environment variable the same way as when the variable is unset. ## Special configuration types -**Status**: [Stable](../document-status.md) - ### Boolean value Any value that represents a Boolean MUST be set to true only by the case-insensitive string `"true"`, meaning `"True"` or `"TRUE"` are also accepted, as true. @@ -47,9 +73,9 @@ Renaming or changing the default value MUST NOT happen without a major version u If an implementation chooses to support an integer-valued environment variable, it SHOULD support nonnegative values between 0 and 2³¹ − 1 (inclusive). Individual SDKs MAY choose to support a larger range of values. > The following paragraph was added after stabilization and the requirements are -thus qualified as "SHOULD" to allow implementations to avoid breaking changes. -For new -implementations, these should be treated as MUST requirements. +> thus qualified as "SHOULD" to allow implementations to avoid breaking changes. +> For new +> implementations, these should be treated as MUST requirements. For variables accepting a numeric value, if the user provides a value the implementation cannot parse, or which is outside the valid range for the configuration item, the implementation SHOULD @@ -83,8 +109,6 @@ For example, the value `12000` indicates 12000 milliseconds, i.e., 12 seconds. ## General SDK Configuration -**Status**: [Stable](../document-status.md) - | Name | Description | Default | Notes | |--------------------------|---------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | OTEL_SDK_DISABLED | Disable the SDK for all signals | false | Boolean value. If "true", a no-op SDK implementation will be used for all telemetry signals. Any other value or absence of the variable will have no effect and the SDK will remain enabled. This setting has no effect on propagators configured through the OTEL_PROPAGATORS variable. | @@ -129,8 +153,6 @@ Depending on the value of `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` may b ## Batch Span Processor -**Status**: [Stable](../document-status.md) - | Name | Description | Default | Notes | | ------------------------------ | ---------------------------------------------------------------- | ------- | ----------------------------------------------------- | | OTEL_BSP_SCHEDULE_DELAY | Delay interval (in milliseconds) between two consecutive exports | 5000 | | @@ -140,8 +162,6 @@ Depending on the value of `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` may b ## Batch LogRecord Processor -**Status**: [Stable](../document-status.md) - | Name | Description | Default | Notes | | ------------------------------- | ---------------------------------------------------------------- | ------- | ------------------------------------------------------ | | OTEL_BLRP_SCHEDULE_DELAY | Delay interval (in milliseconds) between two consecutive exports | 1000 | | @@ -163,8 +183,6 @@ See the SDK [Attribute Limits](../common/README.md#attribute-limits) section for ## Span Limits -**Status**: [Stable](../document-status.md) - See the SDK [Span Limits](../trace/sdk.md#span-limits) section for the definition of the limits. | Name | Description | Default | Notes | @@ -178,8 +196,6 @@ See the SDK [Span Limits](../trace/sdk.md#span-limits) section for the definitio ## LogRecord Limits -**Status**: [Stable](../document-status.md) - See the SDK [LogRecord Limits](../logs/sdk.md#logrecord-limits) section for the definition of the limits. | Name | Description | Default | Notes | @@ -193,8 +209,6 @@ See [OpenTelemetry Protocol Exporter Configuration Options](../protocol/exporter ## Zipkin Exporter -**Status**: [Stable](../document-status.md) - | Name | Description | Default | | ----------------------------- | ---------------------------------------------------------------------------------- |------------------------------------- | | OTEL_EXPORTER_ZIPKIN_ENDPOINT | Endpoint for Zipkin traces | `http://localhost:9411/api/v2/spans` | @@ -220,8 +234,6 @@ _is no specified default, or configuration via environment variables_. ## Exporter Selection -**Status**: [Stable](../document-status.md) - We define environment variables for setting one or more exporters per signal. | Name | Description | Default | @@ -260,8 +272,6 @@ NOT be supported by new implementations. ## Metrics SDK Configuration -**Status**: [Mixed](../document-status.md) - ### Exemplar **Status**: [Experimental](../document-status.md) @@ -278,8 +288,6 @@ Known values for `OTEL_METRICS_EXEMPLAR_FILTER` are: ### Periodic exporting MetricReader -**Status**: [Stable](../document-status.md) - Environment variables specific for the push metrics exporters (OTLP, stdout, in-memory) that use [periodic exporting MetricReader](../metrics/sdk.md#periodic-exporting-metricreader). @@ -288,6 +296,46 @@ that use [periodic exporting MetricReader](../metrics/sdk.md#periodic-exporting- | `OTEL_METRIC_EXPORT_INTERVAL` | The time interval (in milliseconds) between the start of two export attempts. | 60000 | | | `OTEL_METRIC_EXPORT_TIMEOUT` | Maximum allowed time (in milliseconds) to export data. | 30000 | | +## File Configuration + +**Status**: [Experimental](../document-status.md) + +Environment variables involved in [file configuration](file-configuration.md). + +| Name | Description | Default | Notes | +|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|-----------| +| OTEL_EXPERIMENTAL_CONFIG_FILE | The path of the configuration file used to configure the SDK. If set, the configuration in this file takes precedence over all other SDK configuration environment variables. | | See below | + +If `OTEL_EXPERIMENTAL_CONFIG_FILE` is set, the file at the specified path is used to +call [Parse](file-configuration.md#parse). The +resulting [configuration model](./file-configuration.md#configuration-model) is +used to call [Create](file-configuration.md#create) to produce fully configured +SDK components. + +When `OTEL_EXPERIMENTAL_CONFIG_FILE` is set, all other environment variables +besides those referenced in the configuration file +for [environment variable substitution](file-configuration.md#environment-variable-substitution) +MUST be ignored. Ignoring the environment variables is necessary because +there is no intuitive way to merge the flat environment variable scheme with the +structured file configuration scheme in all cases. Users that require merging +multiple sources of configuration are encouraged to customize the configuration +model returned by `Parse` before `Create` is called. For example, a user may +call `Parse` on multiple files and define logic from merging the resulting +configuration models, or overlay values from environment variables on top of a +configuration model. Implementations MAY provide a mechanism to customize the +configuration model parsed from `OTEL_EXPERIMENTAL_CONFIG_FILE`. + +Users are encouraged to use the `sdk-config.yaml` (TODO: Add link when +available) as a starting point for `OTEL_EXPERIMENTAL_CONFIG_FILE`. This file +represents a common SDK configuration scenario, and includes environment +variable substitution references to environment variables which are otherwise +ignored. + +TODO: deprecate env vars which are not +compatible ([#3967](https://github.com/open-telemetry/opentelemetry-specification/issues/3967)) +TODO: provide solution for platforms to contribute to +configure ([#3966](https://github.com/open-telemetry/opentelemetry-specification/issues/3966)) + ## Language Specific Environment Variables To ensure consistent naming across projects, this specification recommends that language specific environment variables are formed using the following convention: