Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Added documentation for the plugins.query.field_type_tolerance setting (#1300) #3143

Merged
merged 1 commit into from
Nov 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions docs/user/admin/settings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -824,3 +824,61 @@ To Re-enable Data Sources:::
}
}

plugins.query.field_type_tolerance
==================================

Description
-----------

This setting controls whether preserve arrays. If this setting is set to false, then an array is reduced
to the first non array value of any level of nesting.

1. The default value is true (preserve arrays)
2. This setting is node scope
3. This setting can be updated dynamically

Querying a field containing array values will return the full array values::

os> SELECT accounts FROM people;
fetched rows / total rows = 1/1
+-----------------------+
| accounts |
+-----------------------+
| [{'id': 1},{'id': 2}] |
+-----------------------+

Disable field type tolerance::

>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_plugins/_query/settings -d '{
"transient" : {
"plugins.query.field_type_tolerance" : false
}
}'

When field type tolerance is disabled, arrays are collapsed to the first non array value::

os> SELECT accounts FROM people;
fetched rows / total rows = 1/1
+-----------+
| accounts |
+-----------+
| {'id': 1} |
+-----------+

Reenable field type tolerance::

>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_plugins/_query/settings -d '{
"transient" : {
"plugins.query.field_type_tolerance" : true
}
}'

Limitations:
------------
OpenSearch does not natively support the ARRAY data type but does allow multi-value fields implicitly. The
SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch
responses, it expects data to match the declared type and does not account for data in array format. If the
plugins.query.field_type_tolerance setting is enabled, the SQL/PPL plugin will handle array datasets by returning
scalar data types, allowing basic queries (e.g., SELECT * FROM tbl WHERE condition). However, using multi-value
fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the
first element of an array is returned, preserving the default behavior.
29 changes: 29 additions & 0 deletions docs/user/limitations/limitations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,32 @@ The response in JDBC format with cursor id::
}

The query with `aggregation` and `join` does not support pagination for now.

Limitations on Using Multi-valued Fields
========================================

OpenSearch does not natively support the ARRAY data type but does allow multi-value fields implicitly. The
SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch
responses, it expects data to match the declared type and does not account for data in array format. If the
plugins.query.field_type_tolerance setting is enabled, the SQL/PPL plugin will handle array datasets by returning
scalar data types, allowing basic queries (e.g., SELECT * FROM tbl WHERE condition). However, using multi-value
fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the
first element of an array is returned, preserving the default behavior.

For example, the following query tries to calculate the absolute value of a field that contains arrays of
longs::

POST _plugins/_sql/
{
"query": "SELECT id, ABS(long_array) FROM multi_value_long"
}
The response in JSON format is::

{
"error": {
"reason": "Invalid SQL query",
"details": "invalid to get longValue from value of type ARRAY",
"type": "ExpressionEvaluationException"
},
"status": 400
}
5 changes: 5 additions & 0 deletions doctest/test_data/multi_value_long.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{"id": 1, "long_array": [1, 2]}
{"id": 2, "long_array": [3, 4]}
{"id": 3, "long_array": [1, 5]}
{"id": 4, "long_array": [1, 2]}
{"id": 5, "long_array": [2, 3]}
Loading