docs: add Troubleshooting and Expected numbers for VictoriaMetrics

Update docs for VictoriaMetrics to point out possible bottlenecks
and provide expected numbers.

docs/victoriametrics.md

@@ -120,3 +120,55 @@ Comma-separated list of URLs to connect to for querying. It can be
 just a single-version URL or list of VMSelect URLs. Workers will be
 distributed in a round robin fashion across the URLs. See help for additional info.
 
+## Troubleshooting
+
+For stable and optimal performance it is recommended running `tsbs_run_queries_victoriametrics`
+or `tsbs_load_victoriametrics` utilities on a separate instance located in the same network
+as VictoriaMetrics installation. This will ensure that neither benchmark utilities nor
+VictoriaMetrics components won't interfere for resources with each other.
+
+In order understand the bottlenecks it is recommended to have [monitoring](https://docs.victoriametrics.com/Quick-Start.html#monitoring)
+in place for the VictoriaMetrics components and also host systems. Please note, that in order to monitor cluster during
+the benchmark it is better to use a separate instance of VictoriaMetrics. You can set up a VMSingle on a separate 
+machine or on machine which is generating load. This will allow to separate benchmark and monitoring loads during the test.
+
+## Expected numbers
+
+__Updated at 15.08.2023__
+
+For a baseline we conducted a benchmark with the following configuration:
+* 3 nodes of vmstorage, vminsert, vmselect located on 3 VMs (1 instance of each per machine) 
+4 CPU / 16GB (GCP e2-standard-4 machines), 100GB pd-balanced disks;
+* load sent from separate machine of the same type (4 CPU / 16 GB);
+* load node had nginx installed to load balance requests between all insert nodes.
+
+Data was generated by the following command:
+```text
+FORMATS=victoriametrics SCALE=1000 TS_START=2023-08-01T00:00:00Z TS_END=2021-08-10T00:00:00Z  ./scripts/generate_data.sh
+```
+
+Load was sent by using the following command:
+```text
+cat /tmp/bulk_data/victoriametrics-data.gz | gunzip | tsbs_load_victoriametrics --urls="http://localhost:8480/insert/0/influx/write" --workers=16 --batch-size=100000
+```
+
+The numbers during the benchmark were the following:
+```text
+loaded 777600000 metrics in 430.052sec with 16 workers (mean rate 1808153.94 metrics/sec)
+loaded 77760000 rows in 430.052sec with 16 workers (mean rate 180815.39 rows/sec)
+```
+This setup utilized 100% CPU and around 2.6GB of memory during the TSBS run.
+
+The same setup with write and read loads running concurrently has the following results:
+* ingestion rate - 1.2M datapoints per second;
+* read requests - 8 RPS.
+
+TSBS report:
+```text
+  Run complete after 1000 queries with 16 workers (Overall query rate 8.86 queries/sec):
+  VictoriaMetrics max of all CPU metrics, random    8 hosts, random 8h0m0s by 1h:
+  min:    56.71ms, med:   799.07ms, mean:  1780.08ms, max: 14641.15ms, stddev:  2245.21ms, sum: 1780.1sec, count: 1000
+  all queries:
+  min:    56.71ms, med:   799.07ms, mean:  1780.08ms, max: 14641.15ms, stddev:  2245.21ms, sum: 1780.1sec, count: 1000
+  wall clock time: 112.885134sec
+```