Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ElasticSearch Exporter does not follow ECS guidelines and logging reference #3091

Open
ShadowMaxLeb opened this issue Oct 11, 2024 · 3 comments · May be fixed by #3107
Open

ElasticSearch Exporter does not follow ECS guidelines and logging reference #3091

ShadowMaxLeb opened this issue Oct 11, 2024 · 3 comments · May be fixed by #3107
Labels
bug Something isn't working good first issue Good for newcomers help wanted Good for taking. Extra help will be provided by maintainers triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@ShadowMaxLeb
Copy link
Contributor

Describe your environment

Version 1.17.0 of this project.

Steps to reproduce
Use the ElasticsearchLogRecordExporter

What is the expected behavior?
The json sent should follow the ECS guidelines as much as possible as well as the ECS logging reference

What is the actual behavior?
Some basic field needed by ELK are not there like @timestamp (timestamp today in the code)

General guidelines
The document MUST have the @timestamp field.
Use the [data types](https://www.elastic.co/guide/en/elasticsearch/reference/8.15/mapping-types.html) defined for an ECS field.
Use the ecs.version field to define which version of ECS is used.
Map as many fields as possible to ECS.

All attributes/resources are set under the attributes/resources key while I think they should be set at the root of the json or at least give a possibility to set attributes/resources at the root of the object so we can fill some fields like user.id
@ShadowMaxLeb ShadowMaxLeb added the bug Something isn't working label Oct 11, 2024
@github-actions github-actions bot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Oct 11, 2024
@marcalff marcalff added triage/accepted Indicates an issue or PR is ready to be actively worked on. help wanted Good for taking. Extra help will be provided by maintainers and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 17, 2024
@github-actions GitHub Actions
Copy link

This issue is available for anyone to work on. Make sure to reference this issue in your pull request.
✨ Thank you for your contribution! ✨

@marcalff marcalff added the good first issue Good for newcomers label Oct 17, 2024
@marcalff marcalff mentioned this issue Oct 17, 2024
Open
@Neema-Joju
Copy link

Hi,
I am an outreachy applicant. May I attempt this?

@ShadowMaxLeb
Copy link
Contributor Author

I have started some work that I should share soon.
There might be some point to discuss.

ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search log message within message key instead of
9cbd5d0 
`body`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-message

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set severity within log.level key inste…
958352a 
…ad of

    `severity`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-level

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set timestamp within @timestamp instead of
2921f56 
`timestamp`

Also changes the format to be a Date string.

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-timestamp

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set instrumentation scope within `log.log…
b03c5d0 
…ger` instead of

`name`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-logger

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
EXPORTERS]: elastic search recorable constructor sets ecs.version f…
4263943 
…ield

to 8.11.0

According to ECS guidelines this field is mandatory
https://www.elastic.co/guide/en/ecs/8.11/ecs-guidelines.html

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search put attributes in json root instead of under
6eb82ab 
`attributes`

This allows user to set other fields that are part of the [ECS log documentation](https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html).
For instance, it allows to have an attribute with key `log.file`, that
will, thanks to `nlohmann::json`, appear as :
```
{
  "log": {
    "file": "xxx"
  }
}
```

Closes open-telemetry#3091
@ShadowMaxLeb ShadowMaxLeb linked a pull request Oct 18, 2024 that will close this issue
Open
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set timestamp within @timestamp instead of
c6c19ef 
`timestamp`

Also changes the format to be a Date string.

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-timestamp

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set instrumentation scope within `log.log…
0b2abbd 
…ger` instead of

`name`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-logger

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
EXPORTERS]: elastic search recorable constructor sets ecs.version f…
f8f0c57 
…ield

to 8.11.0

According to ECS guidelines this field is mandatory
https://www.elastic.co/guide/en/ecs/8.11/ecs-guidelines.html

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search put attributes in json root instead of under
c295345 
`attributes`

This allows user to set other fields that are part of the [ECS log documentation](https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html).
For instance, it allows to have an attribute with key `log.file`, that
will, thanks to `nlohmann::json`, appear as :
```
{
  "log": {
    "file": "xxx"
  }
}
```

Closes open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set timestamp within @timestamp instead of
9a28a01 
`timestamp`

Also changes the format to be a Date string.

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-timestamp

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set instrumentation scope within `log.log…
b644f21 
…ger` instead of

`name`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-logger

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
EXPORTERS]: elastic search recorable constructor sets ecs.version f…
2ecc349 
…ield

to 8.11.0

According to ECS guidelines this field is mandatory
https://www.elastic.co/guide/en/ecs/8.11/ecs-guidelines.html

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search put attributes in json root instead of under
67d7e6b 
`attributes`

This allows user to set other fields that are part of the [ECS log documentation](https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html).
For instance, it allows to have an attribute with key `log.file`, that
will, thanks to `nlohmann::json`, appear as :
```
{
  "log": {
    "file": "xxx"
  }
}
```

Closes open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search log message within message key instead of
3c450ef 
`body`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-message

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set severity within log.level key inste…
8be43d2 
…ad of

    `severity`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-level

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set timestamp within @timestamp instead of
53d72bf 
`timestamp`

Also changes the format to be a Date string.

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-base.html#field-timestamp

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search set instrumentation scope within `log.log…
ff35e7a 
…ger` instead of

`name`

According to ECS logging reference
https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html#field-log-logger

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
EXPORTERS]: elastic search recorable constructor sets ecs.version f…
401ad81 
…ield

to 8.11.0

According to ECS guidelines this field is mandatory
https://www.elastic.co/guide/en/ecs/8.11/ecs-guidelines.html

Refs open-telemetry#3091
ShadowMaxLeb added a commit to Shadow-Official/opentelemetry-cpp that referenced this issue Oct 18, 2024
@ShadowMaxLeb
[EXPORTERS]: elastic search put attributes in json root instead of under
6d72624 
`attributes`

This allows user to set other fields that are part of the [ECS log documentation](https://www.elastic.co/guide/en/ecs/8.11/ecs-log.html).
For instance, it allows to have an attribute with key `log.file`, that
will, thanks to `nlohmann::json`, appear as :
```
{
  "log": {
    "file": "xxx"
  }
}
```

Closes open-telemetry#3091
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working good first issue Good for newcomers help wanted Good for taking. Extra help will be provided by maintainers triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Elastic Search exporter follow ECS guidelines Shadow-Official/opentelemetry-cpp
3 participants
@marcalff @Neema-Joju @ShadowMaxLeb