Publish throughput drop if the size of the stream reaches ~500MB [v2.10.18]

[s4iko]

Observed behavior

I deployed a Nats cluster using the helm chart.
3 servers with limits:

• 2 CPU
• 4Gi memory

I added 2 streams:

• 1 with a file storage (discard old, limits 10M messages)
• 1 with a memory storage (discard old, limits 1M messages)

When I publish messages to the streams it perform well at start:

╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                Streams                                               │
├──────────────────────────────┬─────────────┬─────────────────────┬──────────┬─────────┬──────────────┤
│ Name                         │ Description │ Created             │ Messages │ Size    │ Last Message │
├──────────────────────────────┼─────────────┼─────────────────────┼──────────┼─────────┼──────────────┤
│ stream-file                  │             │ 2023-11-22 18:17:43 │ 838,903  │ 145 MiB │ 5.59s        │

nats bench --js --pub 1 --msgs 100000 --size 128 --syncpub --no-progress --stream "stream-file" "stream-file.subject"
Pub stats: 3,226 msgs/sec ~ 403.30 KB/sec

but after the size of the stream reaches ~500MB the througput drop significantly:

╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                Streams                                                │
├──────────────────────────────┬─────────────┬─────────────────────┬───────────┬─────────┬──────────────┤
│ Name                         │ Description │ Created             │ Messages  │ Size    │ Last Message │
├──────────────────────────────┼─────────────┼─────────────────────┼───────────┼─────────┼──────────────┤
│ stream-file                  │             │ 2023-11-22 18:17:43 │ 3,000,000 │ 518 MiB │ 5.98s        │

nats bench --js --pub 1 --msgs 100000 --size 128 --syncpub --no-progress --stream "stream-file" "stream-file.subject"
Pub stats: 630 msgs/sec ~ 78.76 KB/sec

The same happens for both file or memory streams.
The same happens if no message limit is set.

Thank you for your help (and your awsome product) :)

Expected behavior

The throughput is stable or at least don't drop significantly.

Server and client version

Image server nats:2.10.18-alpine

nats-box:0.14.1 (nats cli v0.1.1)

Host environment

I deployed a Nats cluster using the helm chart.

3 servers with limits:

• 2 CPU
• 3Gi memory

I added 2 streams:

• 1 with a file storage
• 1 with a memory storage

Steps to reproduce

Create the stream:

nats stream add \
  --retention=limits \
  --storage=file \
  --replicas=1 \
  --discard=old \
  --dupe-window=2m \
  --max-age=-1 \
  --max-msgs=1000000 \
  --max-bytes=-1 \
  --max-msg-size=-1 \
  --max-msgs-per-subject=-1 \
  --max-consumers=-1 \
  --allow-rollup \
  --no-deny-delete \
  --no-deny-purge \
  --subjects="stream-file.subject" \
  "stream-file" > /dev/null

Check the throughput when the stream is empty:

nats bench --js --pub 1 --msgs 100000 --size 1024 --syncpub --no-progress --stream "stream-file" "stream-file.subject"

Publish enough messages to reach ~500MB to 600MB, then the next publish througput drops:

nats bench --js --pub 10 --msgs 400000 --size 1024 --syncpub --no-progress --stream "stream-file" "stream-file.subject"

Check that the throughput drops for next publish:

nats bench --js --pub 1 --msgs 100000 --size 1024 --syncpub --no-progress --stream "stream-file" "stream-file.subject"