Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Translate Plugin - Added functionality to provide full path to the source field #3050

Merged

Conversation

vishalboin
Copy link
Contributor

@vishalboin vishalboin commented Jul 21, 2023

Description

Removed the iterateOn option. Now the iterateOn functionality will be taken care by source option with the help of full path to the source.

Configuring source option with path

  • The source provided can be full path to the field in the log that requires translation.

  • If the source path is something like field1/field2/source, all the objects that satisfy the path will be translated based on the provided mappings and the targets fields will be placed in the object along the path field1/field2/target.

  • The fields (field1, field2) leading up to the source field in the path must be JSON arrays or JSON objects. It is important to note that other JSON data types cannot be iterated in this context.

  • The translate_when expression if configured will only be applied for the JSON objects with reference to the path field1/field2.

  • As source option supports multiple sources to be configured, it is required for all the sources in the array to have a common root path.

    mappings:
      - source: ["/field1/field2/source1", "field1/field2/source2"]
        targets:
          - target: "targetField"
             map:
              foo: bar
              

    Let the pipeline configuration be:

    processor:
    - translate:
        mappings:
          - source: collection/status
            targets:
              - target: result
                map:
                  foo: bar
          - source: collection/status/http
            targets:
              - target: result
                map:
                  120: "success"
                  404: "failure"

    Let the contents of logs_json.log be:

    {
      "collection": [
        {
          "status": "foo"
        },
        {
          "status": [
            {
              "http": 120
            }
          ]
        }
      ]
    }

    The translated log would look like this:

    {
      "collection": [
        {
          "status": "foo",
          "result": "bar"
        },
        {
          "status": [
            {
              "http": 120,
              "result": "success"
            }
          ]
        }
      ]
    }
    

Issues Resolved

#1914

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vishal Boinapalli <[email protected]>
@vishalboin vishalboin changed the title Translate Plugin - Removed IterateOn option Translate Plugin - Added functionality to provide full path to the source field Jul 21, 2023
return false;
}
return sourceList.stream().allMatch(sourceItem -> sourceItem instanceof String)
&& commonRootPath(sourceList);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason all the source keys have to have a common root path? If not we should allow different source keys. Can be in future PR

@kkondaka kkondaka merged commit 385dc33 into opensearch-project:main Jul 28, 2023
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants