AWS Environment and performance questions

[aledbf]

👋 Quick question, what's the expected performance using Vitastor in AWS with local NVMEs?

For instance, using d3en.xlarge instances I get this numbers:

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/nbd0
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][32.1%][eta 02m:11s]
test: (groupid=0, jobs=1): err= 0: pid=5630: Thu Jul 13 17:03:31 2023
  write: IOPS=13, BW=54.7MiB/s (57.4MB/s)(3404MiB/62225msec); 0 zone resets
    slat (usec): min=111, max=2743, avg=551.34, stdev=445.80
    clat (msec): min=43, max=4258, avg=2336.55, stdev=541.18
     lat (msec): min=44, max=4259, avg=2337.10, stdev=541.21
    clat percentiles (msec):
     |  1.00th=[  625],  5.00th=[ 1687], 10.00th=[ 1838], 20.00th=[ 2056],
     | 30.00th=[ 2165], 40.00th=[ 2232], 50.00th=[ 2299], 60.00th=[ 2366],
     | 70.00th=[ 2467], 80.00th=[ 2567], 90.00th=[ 3071], 95.00th=[ 3272],
     | 99.00th=[ 3977], 99.50th=[ 4111], 99.90th=[ 4245], 99.95th=[ 4245],
     | 99.99th=[ 4245]
   bw (  KiB/s): min= 8192, max=122880, per=100.00%, avg=59977.14, stdev=19769.18, samples=112
   iops        : min=    2, max=   30, avg=14.64, stdev= 4.83, samples=112
  lat (msec)   : 50=0.12%, 250=0.82%, 750=0.94%, 2000=15.86%, >=2000=82.26%
  cpu          : usr=0.33%, sys=0.07%, ctx=994, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.2%, 4=0.5%, 8=0.9%, 16=1.9%, 32=96.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,851,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=54.7MiB/s (57.4MB/s), 54.7MiB/s-54.7MiB/s (57.4MB/s-57.4MB/s), io=3404MiB (3569MB), run=62225-62225msec

Disk stats (read/write):
  nbd0: ios=51/589, merge=0/286, ticks=1847/1381521, in_queue=1383369, util=99.96%

which is slow.

OS: Debian 12 - no changes
Kernel: 6.1.0-10-cloud-amd64

do I need to tune sysctl?

Thanks!

[vitalif]

Hi, sorry, it seems I missed your issue, as well as several more :-)
I have no idea about AWS, haven't checked Vitastor there. :-)
First you should benchmark the raw disks with fio. Then, if they have write-back cache (i.e. based on consumer NVMes without capacitors) you should add --disable_data_fsync false during vitastor-disk prepare. Then you should also check the network latency.
These are basic things which amount to the performance of a cluster. Theoretical performance is described here https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/performance/theoretical.en.md
We can benchmark it on AWS if you really want and if you give me ssh access :)