Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Can't restore searchable snapshot #9539

Closed
SolovevMV opened this issue Aug 24, 2023 · 2 comments
Closed

[BUG] Can't restore searchable snapshot #9539

SolovevMV opened this issue Aug 24, 2023 · 2 comments
Labels
bug Something isn't working untriaged

Comments

@SolovevMV
Copy link

Describe the bug
When trying to restore a searchable snapshot as an index, I get an error.

[2023-08-22T15:05:07,816][WARN ][o.o.i.c.IndicesClusterStateService] [opensearch-dev-nodes-0] [applogs-2023-08-22-14-00-restored][0] marking and sending shard failed due to [failed recovery]
org.opensearch.indices.recovery.RecoveryFailedException: [applogs-2023-08-22-14-00-restored][0]: Recovery failed on {opensearch-dev-nodes-0}{Xh0d82hZRlqBDtgnTzDY_Q}{AYUf1KrhTwGwu75P8mZRNw}{opensearch-dev-nodes-0}{10.222.44.47:9300}{dms}{shard_indexing_pressure_enabled=true}
	at org.opensearch.index.shard.IndexShard.lambda$executeRecovery$30(IndexShard.java:3554) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:510) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener.completeWith(ActionListener.java:345) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:113) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2620) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.9.0.jar:2.9.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.index.shard.IndexShardRecoveryException: failed recovery
	... 11 more
Caused by: java.lang.ArithmeticException: long overflow
	at java.lang.Math.addExact(Math.java:903) ~[?:?]
	at org.opensearch.repositories.s3.S3RetryingInputStream.openStream(S3RetryingInputStream.java:121) ~[?:?]
	at org.opensearch.repositories.s3.S3RetryingInputStream.<init>(S3RetryingInputStream.java:100) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.readBlob(S3BlobContainer.java:143) ~[?:?]
	at org.opensearch.index.store.remote.utils.TransferManager.lambda$createIndexInput$1(TransferManager.java:87) ~[opensearch-2.9.0.jar:2.9.0]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at org.opensearch.index.store.remote.utils.TransferManager.createIndexInput(TransferManager.java:83) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.utils.TransferManager$DelayedCreationCachedIndexInput.getIndexInput(TransferManager.java:135) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:72) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:147) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seek(OnDemandBlockIndexInput.java:209) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.seek(OnDemandBlockSnapshotIndexInput.java:28) ~[opensearch-2.9.0.jar:2.9.0]
	at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:533) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:87) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:86) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:103) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:92) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:94) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:774) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:146) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.opensearch.index.engine.ReadOnlyEngine.open(ReadOnlyEngine.java:235) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:147) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.indices.IndicesService.lambda$getEngineFactory$10(IndicesService.java:854) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2340) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2302) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2272) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:630) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:115) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener.completeWith(ActionListener.java:342) ~[opensearch-2.9.0.jar:2.9.0]
	... 8 more
[2023-08-22T15:05:07,824][WARN ][o.o.c.r.a.AllocationService] [opensearch-dev-nodes-1] failing shard [failed shard, shard [applogs-2023-08-22-14-00-restored][0], node[Xh0d82hZRlqBDtgnTzDY_Q], [P], recovery_source[snapshot recovery [20dP23p5QUi3_t7A5z5fug] from aws:snapshot-applogs-2023-08-22-14-00/daz0ZU-iRNehzJsxZK6jbg], s[INITIALIZING], a[id=2OYOplidSUOLF9zwedzoZA], unassigned_info[[reason=NEW_INDEX_RESTORED], at[2023-08-22T15:05:06.193Z], delayed=false, details[restore_source[aws/snapshot-applogs-2023-08-22-14-00]], allocation_status[fetching_shard_data]], message [failed recovery], failure [RecoveryFailedException[[applogs-2023-08-22-14-00-restored][0]: Recovery failed on {opensearch-dev-nodes-0}{Xh0d82hZRlqBDtgnTzDY_Q}{AYUf1KrhTwGwu75P8mZRNw}{opensearch-dev-nodes-0}{10.222.44.47:9300}{dms}{shard_indexing_pressure_enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: NotSerializableExceptionWrapper[arithmetic_exception: long overflow]; ], markAsStale [true]]
org.opensearch.indices.recovery.RecoveryFailedException: [applogs-2023-08-22-14-00-restored][0]: Recovery failed on {opensearch-dev-nodes-0}{Xh0d82hZRlqBDtgnTzDY_Q}{AYUf1KrhTwGwu75P8mZRNw}{opensearch-dev-nodes-0}{10.222.44.47:9300}{dms}{shard_indexing_pressure_enabled=true}
	at org.opensearch.index.shard.IndexShard.lambda$executeRecovery$30(IndexShard.java:3554) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:510) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener.completeWith(ActionListener.java:345) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:113) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2620) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.index.shard.IndexShardRecoveryException: failed recovery
	... 11 more
Caused by: org.opensearch.common.io.stream.NotSerializableExceptionWrapper: arithmetic_exception: long overflow
	at java.lang.Math.addExact(Math.java:903) ~[?:?]
	at org.opensearch.repositories.s3.S3RetryingInputStream.openStream(S3RetryingInputStream.java:121) ~[?:?]
	at org.opensearch.repositories.s3.S3RetryingInputStream.<init>(S3RetryingInputStream.java:100) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.readBlob(S3BlobContainer.java:143) ~[?:?]
	at org.opensearch.index.store.remote.utils.TransferManager.lambda$createIndexInput$1(TransferManager.java:87) ~[opensearch-2.9.0.jar:2.9.0]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at org.opensearch.index.store.remote.utils.TransferManager.createIndexInput(TransferManager.java:83) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.utils.TransferManager$DelayedCreationCachedIndexInput.getIndexInput(TransferManager.java:135) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:72) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:147) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seek(OnDemandBlockIndexInput.java:209) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.seek(OnDemandBlockSnapshotIndexInput.java:28) ~[opensearch-2.9.0.jar:2.9.0]
	at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:533) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:87) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:86) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:103) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:92) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:94) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:774) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:146) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.opensearch.index.engine.ReadOnlyEngine.open(ReadOnlyEngine.java:235) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:147) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.indices.IndicesService.lambda$getEngineFactory$10(IndicesService.java:854) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2340) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2302) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2272) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:630) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:115) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.ActionListener.completeWith(ActionListener.java:342) ~[opensearch-2.9.0.jar:2.9.0]
	... 8 more

To Reproduce
Steps to reproduce the behavior:

  1. Take a snapshot from the index
    curl -sk https://localhost:9200/_snapshot/dev/logs-2023-08-22?wait_for_completion=true -H "Content-Type: application/json" -X PUT -d '{"indices": "logs-2023-08-22","ignore_unavailable": true,"include_global_state": false,"partial": false}'
  2. Delete the original index
  3. Restore index from snapshot
    curl -k 'https://localhost:9200/_snapshot/dev/logs-2023-08-22/_restore?wait_for_completion=true' -H 'Content-Type: application/json' -H 'Authorization: Basic YWRtaW46YWRtaW4=' -d '{ "indices": "*", "rename_pattern":"(.+)", "rename_replacement": "$1-restored", "ignore_unavailable": true, "include_global_state": false, "partial": false, "storage_type":"remote_snapshot" }'

Expected behavior
The index recovered from the snapshot should be mounted as a searchable_snapshot index. And a search is possible by this index.

Plugins

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql
repository-s3

Host/Environment (please complete the following information):

  • OS: linux
  • Version 2.9.0
  • Kubernetes cluster v1.24
@SolovevMV SolovevMV added bug Something isn't working untriaged labels Aug 24, 2023
@dblock dblock changed the title [BUG] Cann't restore searchable snapshot [BUG] Can't restore searchable snapshot Aug 24, 2023
@andrross
Copy link
Member

Thanks @SolovevMV. This is the same issue as #9512. You can follow this issue for a 2.9.1 patch release to fix this. The fix will definitely be in the 2.10 release next month as well.

@dblock
Copy link
Member

dblock commented Aug 28, 2023

I am going to close this as a dup. Thanks @andrross and @SolovevMV.

@dblock dblock closed this as completed Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dblock @andrross @SolovevMV