Time Series

[fstagni]

For DIRAC, we want to monitor (and account) for:

1. the server/component themselves
2. the jobs/pilots/transfer etc. Let's just call this the Grid activities.

Mostly we want to see time series for all of them. For 2 we want to do monitoring and long-term accounting (for e.g. reporting purposes). For 1 there's no need for long term accounting. This discussion is about 2.
For 2, In DIRAC we originally had only the "Accounting System" (a custom system with MySQL backend, with auto re-bucketing to reduce the data points), that was also used for real time monitoring. We then added a "Monitoring System" (with ElasticSearch backend). Both backends have been "bended" to be used as Time Series DBs.

"Real" TSDBs (with InfluxDB as main product) are advertised as solutions for monitoring and accounting. For DiracX we should consider if:

• porting the DIRAC accounting and monitoring systems to DiracX (thus keeping MySQL and ES backends)
• moving to a new system

Of course moving to a new system "feels" better, but

• we first need to demonstrate that such a system exists, and it is easy to create (this is the "fun part")
• At the moment we have important "historical" data in MySQL and Elastic/OpenSearch and that would need to be migrated (this is a strong requirement)

[fstagni]

Few notes:

• OpenSearch introduced data streams as an OpenSearch-solution for time-series data. We should almost probably go for it. While in theory this could be done already in DIRAC, it requires first that all indices have an index template, so all this is probably something for DiracX.
• index policies can be defined also in the code, and added via the API
• On visualization: Grafana is most probably the only tool we want to use for visualization. It would be convenient to define few dashboards within the code and apply them via Grafana REST APIs.

I will create an issue from this discussion once the plan is more defined