Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollups not causing shards failures in discover for index pattern * when they are not hidden #844

Closed
xavier-isecurity opened this issue Jul 3, 2023 · 2 comments
Labels
enhancement New request

Comments

@xavier-isecurity
Copy link

xavier-isecurity commented Jul 3, 2023

Is your feature request related to a problem? Please describe.
Rollups can't do searchs with size different than zero. But when we do a discover search, it has a size defined different than zero. If there are rollups created, and they are not hidden, they will trigger shard exception if we search on discover using "*" pattern.

I need rollups not hidden to control them with ISM policy, but I do use "*" index patterns, and it is returning the shard exception everytime I try to use discover.

Describe the solution you'd like
Since rollups can't do searchs with size different than zero, they should be excluded from discover, even if they are not hidden. But still able to execute ism policies and rollovers.

Describe alternatives you've considered
Another way to achieve my desired result, is that ISM policies could be applied also with aliases and not only index prefixes. And that ISM policies are allowed for hidden rollup indexes.

If there is another way to turn this around, please let me know.

Additional context
Exception on discover:
image

  "took": 420,
  "timed_out": false,
  "_shards": {
    "total": 7,
    "successful": 5,
    "skipped": 3,
    "failed": 2,
    "failures": [
      {
        "shard": 0,
        "index": "rollup_test-000001",
        "node": "-U7StgJ0QBSNOtmWa-9l7Q",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Rollup search must have size explicitly set to 0, but found 500"
        }
      }
    ]
  },
@dblock dblock transferred this issue from opensearch-project/OpenSearch Jul 6, 2023
@bowenlan-amzn
Copy link
Member

If your rollup indexes have certain prefix rollup-, you can create index pattern in the dashboard to not include them.

There's a cluster setting plugins.rollup.search.enabled to disable rollup search function which requires size=0, but if you uses "*", it still doesn't make sense to search rollup indexes along with your normal indexes.

@xavier-isecurity
Copy link
Author

If your rollup indexes have certain prefix rollup-, you can create index pattern in the dashboard to not include them.

There's a cluster setting plugins.rollup.search.enabled to disable rollup search function which requires size=0, but if you uses "*", it still doesn't make sense to search rollup indexes along with your normal indexes.

So it happens that even if I exclude the indexes on the searches it returns errors.

So I did find a turn around to this issue.

I added an alias for all index that matters to me when I search with *, not including rollup in this alias. Problem solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New request
Projects
None yet
Development

No branches or pull requests

2 participants