Fix the ingest rate displayed in the CLI. #4682
Conversation
fulmicoton
force-pushed
the
cli-ingest-rate
branch
3 times, most recently
from
4493938
to
aa56186
Compare
|
I tried it and it seems somehow broken. After a while, I get this: Ingestion rate should be in the order of 17MiB/s, but the number I got never seemed to go below 40MiB/s |
|
oh really? let me check! Thanks for testing it! |
|
if that helps, it seems to only take a few discrete values, a few of which are 51.4, 45.7, 40, 62.8, 57.1, 74.2, that seems to be increments of 5.7 from each other |
fulmicoton
force-pushed
the
cli-ingest-rate
branch
6 times, most recently
from
29ec0bc
to
9235400
Compare
|
@trinity-1686a There was a bug in the rate estimator indeed. I added @guilload to the review. |
fulmicoton
force-pushed
the
cli-ingest-rate
branch
from
9235400
to
cd232a0
Compare
|
the estimator always seems to give a multiple of 5.71 for this workload. This is because it essentially count batches of (quasi-)fixed size, and there aren't that many batches processed over a 2s window. In my case, there are almost always 3 batches processed, so I get 17.1MiB most of the time, and 11.4 or 22.8 when 2 or 4 batches got processed. |
The CLI was showing the average rate since the CLI was started. This PR changes it to an estimation over the last 2 seconds.
The rate estimator was returning inflated results whenever we did not record any work during a bucket period. The fix does two things: - It replace the 1-bit color logic `(bucket_ord / num_buckets) % 2`` to a more generic and more accurate 8 bit hash `bucket_ord % 243`. Collision may only happen if no write targetted this specific bucket for `num_buckets x 243 x bucket_period`. Upon read, it checks for the hash and dismisses buckets that are not matching the hash.
Co-authored-by: trinity-1686a <[email protected]>
fulmicoton
force-pushed
the
cli-ingest-rate
branch
from
e83a71a
to
718553a
Compare
|
I think there are different ways to improve the estimation, but I'll merge this for the bugfix. @trinity-1686a feel free to revisit it. |
The CLI was showing the average rate since the CLI was started.
This PR changes it to an estimation over the last 2 seconds.