Install should not rely upon filesystem state in order to prevent database anihilation

[TSP-RomanHargrave]

I've been working on deploying OpenNMS with Kubernetes, and noticed that the user tables disappear every time I apply an update. Looking at create.sql, it appears that this is because /opt/opennms/etc/configured is missing. This is to be expected because I am using etc-overlay and not persisting the full etc folder, and as such changes to etc will be lost on every generation (read: every change to configuration files, container liveness probe failure, node failure, etc...).

It would be relatively easy to fix this by handling the database initialization entirely with liquibase, which appears to be in use already, or querying the database beforehand.

That having been said, it would also make more sense to write the file to the data directory instead of the etc directory.

[TSP-RomanHargrave]

Looking more at the contents of etc, I suppose this is more of an overarching problem.

[indigo423]

I'll agree, the running configuration. Running it as a container myself, I've noticed the following problems especially with the configuration:

• Runtime configuration is persisted in the file system which makes it hard to distribute configs when you want to separate core processes and the web app in dedicated processes
• There is a version dependency between the configuration and the running container image which requires manual tasks on updates when you not use the etc-overlay mechanism
• Runtime configuration can be changed from the WebUI which are persisted in the filesystem. For some daemons there is just the way to configure them in the file system directly.
• The topology UI persists state in Karaf data directory.

IMHO, One way would be to persist runtime configuration in appropriate storage and maintain schema changes, e.g. PostgreSQL with Liquibase.

[blaisep]

FWIW, as a noob, I have run into some of these problems already because I was using the docker image in my dev setup and I didn't know which data files would persist.
@TSP-RomanHargrave , do you suggest , for now, to avoid the docker image in dev setups?
(I think prod is OK because there are no config changes... ;-}

[indigo423]

@blaisep here is what I do, I mount the ${OPENNMS_HOME}/etc directory to the host system. That way the state of configuration files changed through the WebUI are persisted. To make config merges easier, I mount also the etc-overlay directory to the host system. This contains all configuration files I've changed and can't be changed through the web UI. Candidates are e.g.:

• property files in opennms.properties.d
• service-configuration.xml
• poller-configuration.xml
• categories.xml
• viewsdisplay.xml
• collectd / eventd - I don't use the webui to edit those files
• snmp-graph.properties.d
• data collection configuration directories
  etc.

Of course the way OpenNMS persists "runtime" configurations in the file system is old-school. With the upcoming "Sentinel" project we need a centralised way to store configuration and make them accessible over a network. There is no architecture decision made yet and is in flux.

[TSP-RomanHargrave]

• poller-configuration.xml

This is the one that really drives me insane. Restarting OpenNMS on Kubernetes effectively means redeploying the set. This effectively adds around two minutes (worst case) for the implementation (GKE in this case) to regenerate workloads and reconfigure VMs to the already long OpenNMS startup time.

With respect to data/configuration storage, anything is ultimately on the disk. The shotgun approach with a "bundle of files" is technically more human friendly (because you can directly edit the configuration; however, XML makes this a somewhat moot point), but suffers from inconsistent file placement.

Ultimately, the best bet is just using the database as a K/V store. PostgreSQL has some good features for this like embedded JSON, which is pretty useful when you want to correlate "free-form" data, but still want the ability to search it. I can't see an application for PostgreSQL searching in JSON fields, rather just retrieving them; however, I can certainly see a human at the PostgreSQL shell using that feature to find configuration values.

[indigo423]

Ultimately, the best bet is just using the database as a K/V store. PostgreSQL has some good features for this like embedded JSON, which is pretty useful when you want to correlate "free-form" data, but still want the ability to search it. I can't see an application for PostgreSQL searching in JSON fields, rather just retrieving them; however, I can certainly see a human at the PostgreSQL shell using that feature to find configuration values.

Totally agree with you.