Add mention about collecting rails-load-stats

[jhutar]

Add something like this from Lukas Zapletal @lzap:

The best KPI (i.e. Key Performance Indicator) is request processing duration which can be found in telemetry
data and in production.log. Average duration is not very useful as some
endpoints are very fast but some can be slow, this really depends on the
deployment, e.g. if the customer has hosts with many NICs or disks, fact
endpoints can become slow. Therefore it is recommended to monitor these.

If you have a monitoring solution capable of importing data via
statsd protocol, I recommend to enable statsd telemetry in settings.yaml
and send those packets into the customers monitoring solution. If you do
not have anything, analysis of production.log can be also useful. Our CEE
group maintains a script that does exactly just that:

https://github.com/pmoravec/rails-load-stats

This can be done on a daily basis.

Obviously this needs to be reformatted, tested and include example output. We might also want to mention moninoring this via Logstash or Splunk.

[lzap]

The most reliable telemetry collection at the moment is: enable statsd exporter in settings.yaml, run statsd_exporter and collect the data via Prometheus.

We will work on better monitoring setup via PCP later this year.

[jhutar]

Hello @lzap. Does this your statement above still apply please?

[jhutar]

@Imaanpreet To try the setup above do this:

1. Install https://github.com/prometheus/statsd_exporter to Satellite machine
2. Configure Satellite to send stats to statsd exporter as described at https://community.theforeman.org/t/monitoring-foreman-with-prometheus-via-statsd/21508
3. Make Prometheus to scrape statsd exporter by adding something like following to our perf54 /root/prometheus/targets/cluster.yml

- targets:
    - '<satellite>:9099'
    - '<satellite>:9102'

[lzap]

I tested PCP from RHEL 8.5 this weekend and it works great, it has a native statsd PMDA agent now and latest Grafana. I am going to write article/docs and prepare demo, setup is extremely easy now. JFYI.

[jhutar]

CC @IshanSinghal FYI

[jhutar]

Mine expectation here is that once we use pcp (pmdastatsd as Lukas mentions) to capture these Rails metrics, we can use https://man7.org/linux/man-pages/man1/pcp2graphite.1.html to pipe that data into our Graphite (and so into out Grafana).