Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AckFloor is ahead of the NAK'ed message sequence id [v2.10.17] #5727

Open
TheCheshireFox opened this issue Jul 31, 2024 · 3 comments
Open

AckFloor is ahead of the NAK'ed message sequence id [v2.10.17] #5727

TheCheshireFox opened this issue Jul 31, 2024 · 3 comments
Labels
defect Suspected defect such as a bug or regression stale This issue has had no activity in a while

Comments

@TheCheshireFox
Copy link

Observed behavior

So far, for unknown reasons, the stream and ack floor sequence goes further away than the message sequence. This leads to some messages not being delivered.

In the same time we have this lines in logs:

[3924] 2024/07/25 09:39:20.399780 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.ResolvedStream" took too long: 2.8518496s
[3924] 2024/07/25 09:39:20.399780 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.ResolvedStream" took too long: 2.8518496s
[3924] 2024/07/25 09:39:20.399780 [WRN] [fe80::1e08:e369:ec28:43ff%Ethernet]:62197 - cid:139 - Readloop processing time: 2.8518496s
[3924] 2024/07/25 09:58:46.644653 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.ResolvedStream" took too long: 2.7806246s
[3924] 2024/07/25 09:58:46.645165 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.ResolvedStream" took too long: 2.7811366s
[3924] 2024/07/25 09:58:46.645165 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.ResolvedStream" took too long: 2.7811366s
[3924] 2024/07/25 09:58:46.645165 [WRN] [fe80::1e08:e369:ec28:43ff%Ethernet]:62197 - cid:139 - Readloop processing time: 2.7811366s
[3924] 2024/07/25 11:48:44.800632 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.SessionEventsStream" took too long: 19.7930954s
[3924] 2024/07/25 11:48:44.800632 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.SessionEventsStream" took too long: 19.7930954s
[3924] 2024/07/25 11:48:44.800632 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.SessionEventsStream" took too long: 19.7930954s
[3924] 2024/07/25 11:48:44.800632 [WRN] [fe80::1e08:e369:ec28:43ff%Ethernet]:62255 - cid:142 - Readloop processing time: 19.7930954s
[3924] 2024/07/25 12:06:30.730378 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:06:55.687925 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:07:18.684606 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:08:58.826814 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:22.349081 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:41.269744 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:55.141511 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:55.173560 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:55.191331 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:55.208208 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.205579 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.219878 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:59.234684 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:59.245033 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.257238 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.269476 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:59.280122 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.288702 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.298228 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.311782 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.341952 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:59.366353 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.389319 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.429338 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.447628 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410384] out of range
[3924] 2024/07/25 12:09:59.476263 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.602504 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.620547 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:09:59.634582 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.644213 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.658049 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.667945 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.678536 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.696812 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.709234 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.728542 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.739676 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.752390 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.766410 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.791476 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.804872 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.820241 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.834459 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:09:59.851933 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:09:59.901653 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:10:02.197363 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:10:29.738856 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:10:30.277279 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:10:30.926567 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:10:32.068136 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:10:33.121954 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:10:33.147361 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:10:33.166727 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:10:37.984535 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410384] out of range
[3924] 2024/07/25 12:10:38.006142 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410200] out of range
[3924] 2024/07/25 12:10:38.046319 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:10:38.440940 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:10:38.570226 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410203] out of range
[3924] 2024/07/25 12:10:38.599436 [WRN] Consumer '$G > ProcessingStream > ProcessorService_ProcessingStream_Consumer_b7995ccd001f4c7aa2a33473006cb572_High' error on write store state from purge: bad pending entry, sequence [410369] out of range
[3924] 2024/07/25 12:43:14.688611 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.SessionEventsStream" took too long: 12.385257s
[3924] 2024/07/25 12:43:14.688611 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.SessionEventsStream" took too long: 12.385257s
[3924] 2024/07/25 12:43:14.688611 [WRN] Internal subscription on "$JS.API.STREAM.PURGE.SessionEventsStream" took too long: 12.385257s
[3924] 2024/07/25 12:43:14.688611 [WRN] [fe80::1e08:e369:ec28:43ff%Ethernet]:62255 - cid:142 - Readloop processing time: 12.385257s

Expected behavior

All messages in a stream must be delivered.

Server and client version

nats-server: v2.10.17
dotnet nats-client: 1.0.4

Host environment

os: Windows Server 2019 Standard
arch: x64
ram: 6 Gb

Steps to reproduce

I'm not sure what the steps to reproduce are, but we got such a problem for the message we NAK on the stream we actively purging.
@TheCheshireFox TheCheshireFox added the defect Suspected defect such as a bug or regression label Jul 31, 2024
@wallyqs
Copy link
Member

wallyqs commented Jul 31, 2024

@TheCheshireFox could you share an example of how you are purging the stream?

@TheCheshireFox
Copy link
Author

Sure. We use this method:

public ulong PurgeStream(string streamName, string subject, CancellationToken cancellationToken)
{
    var optionsBuilder = new PurgeOptions.PurgeOptionsBuilder();
    optionsBuilder.WithSubject(subject);
    var response = JetStreamManagement.PurgeStream(streamName, optionsBuilder.Build()); // JetStreamManagement has a type IJetStreamManagement
    return response.Purged;
}

@TheCheshireFox
Copy link
Author

@wallyqs, any news? Maybe I can provide more information?

@wallyqs wallyqs changed the title AckFloor is ahead of the NAK'ed message sequence id AckFloor is ahead of the NAK'ed message sequence id [v2.10.17] Sep 5, 2024
@github-actions github-actions bot added the stale This issue has had no activity in a while label Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
defect Suspected defect such as a bug or regression stale This issue has had no activity in a while
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wallyqs @TheCheshireFox