Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Missing privileges in FileCacheFacroty and TransferManager on search that includes searchable snapshots #16542

Closed
asor-gl opened this issue Nov 1, 2024 · 3 comments
Closed

[BUG] Missing privileges in FileCacheFacroty and TransferManager on search that includes searchable snapshots #16542

asor-gl opened this issue Nov 1, 2024 · 3 comments
Labels
bug Something isn't working Search:Searchable Snapshots v2.19.0 Issues and PRs related to version 2.19.0 v3.0.0 Issues and PRs related to version 3.0.0

Comments

@asor-gl
Copy link

asor-gl commented Nov 1, 2024

Describe the bug

When a search is preformed that includes searchable snapshots it fails with runtime error. Added trace can be examined to see more detail about missing permissions to perform file interaction.

Related component

Search:Searchable Snapshots

To Reproduce

  1. Perform a search on indices that include searchable snapshots.
  2. Search results in error returning runtime error.

Expected behavior

Search is performed successfully.

Additional Details

Plugins

Screenshots
Error traces where missing permissions are found:

opensearch-master-data-1 opensearch [2024-10-26T22:23:45,806][TRACE][o.o.s.SearchService      ] [opensearch-master-data-1] Query phase failed
opensearch-master-data-1 opensearch org.opensearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:307) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:155) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1850) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1911) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:345) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:328) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.lambda$computeIfAbsent$0(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.Cache.compute(Cache.java:483) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:434) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.computeIfAbsent(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:288) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1917) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1848) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:641) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:707) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:676) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
opensearch-master-data-1 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
opensearch-master-data-1 opensearch 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
opensearch-master-data-1 opensearch Caused by: org.opensearch.script.ScriptException: runtime error
opensearch-master-data-1 opensearch 	at org.opensearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:109) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('streams') && doc['streams'].size() > 0 ...:1) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	... 26 more
opensearch-master-data-1 opensearch Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/opensearch/data/nodes/0/cache/otDA0XeQT4uO9HBczv6Vpw/0/RemoteLocalStore/_4p_Lucene90_0.dvd_block_1" "delete")
opensearch-master-data-1 opensearch 	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.lang.SecurityManager.checkDelete(SecurityManager.java:861) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/sun.nio.fs.UnixPath.checkDelete(UnixPath.java:803) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:269) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109) ~[?:?]
opensearch-master-data-1 opensearch 	at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$0(FileCacheFactory.java:58) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.ExceptionsHelper.catchAsRuntimeException(ExceptionsHelper.java:352) ~[opensearch-core-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$1(FileCacheFactory.java:58) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.cache.LRUCache.removeNode(LRUCache.java:366) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.cache.LRUCache.remove(LRUCache.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.cache.SegmentedCache.remove(SegmentedCache.java:105) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.filecache.FileCache.remove(FileCache.java:105) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager$DelayedCreationCachedIndexInput.getIndexInput(TransferManager.java:169) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:89) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:160) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seek(OnDemandBlockIndexInput.java:209) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.seek(OnDemandBlockSnapshotIndexInput.java:30) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1122) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.opensearch.index.fielddata.FieldData$12.nextValue(FieldData.java:456) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$BinaryScriptDocValues.setNextDocId(ScriptDocValues.java:514) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$Strings.setNextDocId(ScriptDocValues.java:547) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:103) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:56) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('streams') && doc['streams'].size() > 0 ...:36) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-1 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-1 opensearch 	... 26 more

After following exception was fixed another one was present:

[2024-10-28T16:36:55,861][TRACE][o.o.s.SearchService      ] [opensearch-master-data-0] Query phase failed
opensearch-master-data-0 opensearch org.opensearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:307) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:155) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1850) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1911) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:345) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:328) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.lambda$computeIfAbsent$0(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.Cache.compute(Cache.java:483) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:434) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.computeIfAbsent(OpenSearchOnHeapCache.java:98) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:288) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1917) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1848) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:641) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:707) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:676) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
opensearch-master-data-0 opensearch 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
opensearch-master-data-0 opensearch 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
opensearch-master-data-0 opensearch Caused by: org.opensearch.script.ScriptException: runtime error
opensearch-master-data-0 opensearch 	at org.opensearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:109) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('gl2_source_node') && doc['gl2_source_node'].size() > 0 ...:1) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	... 26 more
opensearch-master-data-0 opensearch Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/opensearch/data/nodes/0/cache/GGfzun4mSpSiqhVfEnhizA/0/RemoteLocalStore/_qs_Lucene90_0.dvd_block_5" "read")
opensearch-master-data-0 opensearch 	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:789) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:563) ~[?:?]
opensearch-master-data-0 opensearch 	at java.base/java.nio.file.Files.exists(Files.java:2514) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager.createIndexInput(TransferManager.java:111) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager$DelayedCreationCachedIndexInput.getIndexInput(TransferManager.java:166) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:89) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:160) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seek(OnDemandBlockIndexInput.java:209) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.seek(OnDemandBlockSnapshotIndexInput.java:30) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1122) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.opensearch.index.fielddata.FieldData$12.nextValue(FieldData.java:456) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$BinaryScriptDocValues.setNextDocId(ScriptDocValues.java:514) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.index.fielddata.ScriptDocValues$Strings.setNextDocId(ScriptDocValues.java:547) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:103) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:56) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.painless.PainlessScript$Script.execute((doc.containsKey('gl2_source_node') && doc['gl2_source_node'].size() > 0 ...:44) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.support.values.ScriptBytesValues.advanceExact(ScriptBytesValues.java:76) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ValuesSourceCollectorSource$1.collect(MapStringTermsAggregator.java:213) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:129) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:115) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.filter.FiltersAggregator$1.collect(FiltersAggregator.java:186) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BestBucketsDeferringCollector.prepareSelectedBuckets(BestBucketsDeferringCollector.java:221) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.DeferableBucketAggregator.beforeBuildingBuckets(DeferableBucketAggregator.java:119) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:217) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:406) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator.buildAggregations(AutoDateHistogramAggregator.java:257) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.bucket.histogram.AutoDateHistogramAggregator$FromSingle.buildAggregations(AutoDateHistogramAggregator.java:439) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:205) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.aggregations.BucketCollectorProcessor.processPostCollection(BucketCollectorProcessor.java:78) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:292) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	at org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61) ~[?:?]
opensearch-master-data-0 opensearch 	at org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282) ~[opensearch-2.17.1.jar:2.17.1]
opensearch-master-data-0 opensearch 	... 26 more

Host/Environment (please complete the following information):

  • OS: docker image: opensearchproject/opensearch:2.17.1
  • Version 2.17.1

Additional context
We tested the build with internal patch adding needed permissions and it fixes the issue, please let me know if i should create a PR for it.
@asor-gl asor-gl added bug Something isn't working untriaged labels Nov 1, 2024
@finnegancarroll
Copy link
Contributor

Hi @asor-gl, thanks for opening this issue.
This looks a lot like #14268 and i'm thinking the fix for that did not go far enough. The previous issue elevated permissions for fileCache.compute since inserting into the cache can trigger evictions.

In this case it seems script execution is trying to access the underlying file associated with a cacheEntry and failing since cacheEntry.getIndexInput() is missing these elevated permissions.

@finnegancarroll finnegancarroll mentioned this issue Nov 1, 2024
Merged
3 tasks
@asor-gl
Copy link
Author

asor-gl commented Nov 2, 2024 •
edited
Loading

@finnegancarroll Thanks for the tip. I'll test if adding elevated permissions higher up will work as well, as in the other issue.

Update: can confirm that #16544 fixes the issue with permissions.

@sandeshkr419
Copy link
Contributor

Closing as #16544 fixes up this issue.

@reta reta added v3.0.0 Issues and PRs related to version 3.0.0 v2.19.0 Issues and PRs related to version 2.19.0 labels Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Search:Searchable Snapshots v2.19.0 Issues and PRs related to version 2.19.0 v3.0.0 Issues and PRs related to version 3.0.0
Projects
Search Project Board
Status: ✅ Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@reta @finnegancarroll @sandeshkr419 @asor-gl