Skip to content

Commit

Permalink
Define file configuration file format and env var substitution (#3744)
Browse files Browse the repository at this point in the history
Part of incorporating [OTEP
#225](open-telemetry/oteps#225) into the
specification.

Followup to #3437.

The adds the YAML file format, and leaves an open TODO for the JSON
format discussed in the original OTEP. It also define environment
variable substitution. IMO its necessary to define environment variable
substitution at the same time as file format because they interact in
ways that aren't immediately obvious and need to be prototyped.

The opentelemetry-java implementation of file configuration already
accepts YAML encoding and performs environment variable substitution as
described in this PR. It does not support JSON, and I don't think we
should unless there is good reason. I omitted the JSON file format
because I don't know of any prototypes that permit it. We should add
JSON once a language implements it and explores how environment variable
substitution works with JSON.
  • Loading branch information
jack-berg authored Dec 14, 2023
1 parent 3d60131 commit e94af89
Showing 1 changed file with 101 additions and 6 deletions.
107 changes: 101 additions & 6 deletions specification/configuration/file-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ linkTitle: File
- [Configuration Model](#configuration-model)
* [Stability Definition](#stability-definition)
- [Configuration file](#configuration-file)
* [YAML file format](#yaml-file-format)
* [Environment variable substitution](#environment-variable-substitution)
- [SDK Configuration](#sdk-configuration)
* [In-Memory Configuration Model](#in-memory-configuration-model)
* [Operations](#operations)
Expand Down Expand Up @@ -46,12 +48,86 @@ TODO: define stability guarantees and backwards compatibility

## Configuration file

A configuration file is a file representation of
A configuration file is a serialized file-based representation of
the [Configuration Model](#configuration-model).

TODO: define acceptable file formats
Configuration files SHOULD use one the following serialization formats:

TODO: define environment variable substitution
### YAML file format

[YAML](https://yaml.org/spec/1.2.2/) configuration files SHOULD follow YAML spec
revision >= 1.2.

YAML configuration files MUST use file extensions `.yaml` or `.yml`.

TODO: decide if JSON file format is required

### Environment variable substitution

Configuration files support environment variables substitution for references
which match the following regular expression:

```regexp
\$\{(?<ENV_NAME>[a-zA-Z_][a-zA-Z0-9_]*)}
```

The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed
by 0 or more alphanumeric or `_` characters.

For example, `${API_KEY}` is valid, while `${1API_KEY}` and `${API_$KEY}` are
invalid.

Environment variable substitution MUST only apply to scalar values. Mapping keys
are not candidates for substitution.

If a referenced environment variable is not defined, it MUST be replaced with an
empty value.

Node types MUST be interpreted after environment variable substitution takes
place. This ensures the environment string representation of boolean, integer,
or floating point fields can be properly converted to expected types.

It MUST NOT be possible to inject YAML structures by environment variables. For
example, references to `INVALID_MAP_VALUE` environment variable below.

For example, consider the following environment variables,
and [YAML](#yaml-file-format) configuration file:

```shell
export STRING_VALUE="value"
export BOOl_VALUE="true"
export INT_VALUE="1"
export FLOAT_VALUE="1.1"
export INVALID_MAP_VALUE"value\nkey:value" # An invalid attempt to inject a map key into the YAML
```

```yaml
string_key: ${STRING_VALUE} # Valid reference to STRING_VALUE
other_string_key: "${STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes
another_string_key: "${BOOl_VALUE}" # Valid reference to BOOl_VALUE inside double quotes
yet_another_string_key: ${INVALID_MAP_VALUE} # Valid reference to INVALID_MAP_VALUE, but YAML structure from INVALID_MAP_VALUE MUST NOT be injected
bool_key: ${BOOl_VALUE} # Valid reference to BOOl_VALUE
int_key: ${INT_VALUE} # Valid reference to INT_VALUE
float_key: ${FLOAT_VALUE} # Valid reference to FLOAT_VALUE
combo_string_key: foo ${STRING_VALUE} ${FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE
undefined_key: ${UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with ""
${STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored
```
Environment variable substitution results in the following YAML:
```yaml
string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
other_string_key: "value" # Interpreted as type string, tag URI tag:yaml.org,2002:str
another_string_key: "true" # Interpreted as type string, tag URI tag:yaml.org,2002:str
yet_another_string_key: "value\nkey:value" # Interpreted as type string, tag URI tag:yaml.org,2002:str
bool_key: true # Interpreted as type bool, tag URI tag:yaml.org,2002:bool
int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int
float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float
combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str
undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null
${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
```
## SDK Configuration
Expand Down Expand Up @@ -81,10 +157,19 @@ with OpAmp

Parse and validate a [configuration file](#configuration-file).

Parse MUST perform [environment variable substitution](#environment-variable-substitution).

Parse MUST interpret null as equivalent to unset.

**Parameters:**

* `file`: The [configuration file](#configuration-file) to parse. This MAY be a
file path, or language specific file data structure, or a stream of a file's content.
* `file_format`: The file format of the `file` (e.g. [yaml](#yaml-file-format)).
Implementations MAY accept a `file_format` parameter, or infer it from
the `file` extension, or include file format specific overloads of `parse`,
e.g. `parseYaml(file)`. If `parse` accepts `file_format`, the API SHOULD be
structured so a user is obligated to provide it.

**Returns:** [configuration model](#in-memory-configuration-model)

Expand All @@ -94,12 +179,19 @@ This SHOULD return an error if:
* The parsed `file` content does not conform to
the [configuration model](#configuration-model) schema.

TODO: define behavior if some portion of configuration model is not supported

#### Create

Interpret [configuration model](#in-memory-configuration-model) and return SDK components.

If a field is null or unset and a default value is defined, Create MUST ensure
the SDK component is configured with the default value. If a field is null or
unset and no default value is defined, Create SHOULD return an error. For
example, if configuring
the [span batching processor](../trace/sdk.md#batching-processor) and
the `scheduleDelayMillis` field is null or unset, the component is configured
with the default value of `5000`. However, if the `exporter` field is null or
unset, Create fails fast since there is no default value for `exporter`.

**Parameters:**

* `configuration` - The configuration model.
Expand All @@ -115,7 +207,10 @@ The multiple responses MAY be returned using a tuple, or some other data
structure encapsulating the components.

This SHOULD return an error if it encounters an error in `configuration` (i.e.
fail fast).
fail fast) in accordance with
initialization [error handling principles](../error-handling.md#basic-error-handling-principles).

TODO: define behavior if some portion of configuration model is not supported

## References

Expand Down

0 comments on commit e94af89

Please sign in to comment.