Issue storing app id's to influx db

[theidemannIf]

Hi,
I've problems to get the app id's stored from Qlik Sense health check into Influx DB. This appears at active docs, loaded docs and in memory docs.
Butler SOS is able to get the count and also able to store the count, but not the id's itselfe.
We are running Butler SOS on Debian 11, Node.js and InfluxDB 1.8 as binary (no docker).

I've tried to find a way to log what's sent by Butler to Influx, but I don't find a way. Does anyone have an idea how to proceed?

Thanks!

[mountaindude]

You can start Butler SOS with the --loglevel verbose or even --loglevel debug options.
That will increase log verbosity.

With verbose log level you get log entries when data has been sent to InfluxDB:

2023-05-05T11:21:15.049Z verbose: --------------------------------
2023-05-05T11:21:15.049Z verbose: Iteration # 4, Uptime: 0 months, 0 days, 0 hours, 0 minutes, 57.038 seconds, Heap used 29.48 MB of total heap 31.67 MB. External (off-heap): 2.7 MB. Memory allocated to process: 89.01 MB.
2023-05-05T11:21:15.122Z verbose: MEMORY USAGE INFLUXDB: Sent Butler SOS memory usage data to InfluxDB
2023-05-05T11:21:18.291Z verbose: PROXY SESSIONS: Event started: Poll user sessions
2023-05-05T11:21:18.311Z verbose: HEALTH: Event started: Statistics collection
2023-05-05T11:21:18.311Z verbose: HEALTH: Getting stats for server: sense1
2023-05-05T11:21:18.314Z verbose: HEALTH: Getting stats for server: sense2
2023-05-05T11:21:18.318Z verbose: APP NAMES: Event started: Get app names
2023-05-05T11:21:18.318Z verbose: APP NAMES: Start getting app names from repository db
2023-05-05T11:21:18.357Z verbose: HEALTH: Received ok response from pro2-win2.lab.ptarmiganlabs.net
2023-05-05T11:21:18.367Z verbose: HEALTH: Received ok response from pro2-win1.lab.ptarmiganlabs.net
2023-05-05T11:21:18.534Z verbose: HEALTH METRICS: Sent health data to Influxdb for server sense2
2023-05-05T11:21:18.590Z verbose: PROXY SESSIONS: Sent user session data to InfluxDB for server "pro2-win2.lab.ptarmiganlabs.net:4243", virtual proxy "/"
2023-05-05T11:21:18.591Z verbose: PROXY SESSIONS: Sent user session data to InfluxDB for server "pro2-win2.lab.ptarmiganlabs.net:4243", virtual proxy "/hdr"
2023-05-05T11:21:18.591Z verbose: PROXY SESSIONS: Sent user session data to InfluxDB for server "pro2-win1.lab.ptarmiganlabs.net:4243", virtual proxy "/sales"
2023-05-05T11:21:18.592Z verbose: HEALTH METRICS: Sent health data to Influxdb for server sense1
2023-05-05T11:21:18.592Z verbose: PROXY SESSIONS: Sent user session data to InfluxDB for server "pro2-win1.lab.ptarmiganlabs.net:4243", virtual proxy "/"
2023-05-05T11:21:18.594Z verbose: PROXY SESSIONS: Sent user session data to InfluxDB for server "pro2-win1.lab.ptarmiganlabs.net:4243", virtual proxy "/hdr"
2023-05-05T11:21:18.605Z verbose: APP NAMES: Number of apps: 329
2023-05-05T11:21:18.609Z verbose: APP NAMES: Done getting app names from repository db
2023-05-05T11:21:18.645Z verbose: PROXY SESSIONS: Sent user session data to InfluxDB for server "pro2-win1.lab.ptarmiganlabs.net:4243", virtual proxy "/sourcing"

With debug log level you get even more data, including the actual data received from Sense:

2023-05-05T11:30:45.364Z verbose: MEMORY USAGE INFLUXDB: Sent Butler SOS memory usage data to InfluxDB
2023-05-05T11:31:00.260Z verbose: --------------------------------
2023-05-05T11:31:00.261Z verbose: Iteration # 2, Uptime: 0 months, 0 days, 0 hours, 0 minutes, 20.034 seconds, Heap used 27.83 MB of total heap 31.22 MB. External (off-heap): 2.54 MB. Memory allocated to process: 86.52 MB.
2023-05-05T11:31:00.261Z debug: MEMORY USAGE: Memory usage {
  "instanceTag": "DEV",
  "heapUsedMByte": 27.83,
  "heapTotalMByte": 31.22,
  "externalMemoryMByte": 2.54,
  "processMemoryMByte": 86.52
})
2023-05-05T11:31:00.510Z verbose: HEALTH: Event started: Statistics collection
2023-05-05T11:31:00.511Z verbose: HEALTH: Getting stats for server: sense1
2023-05-05T11:31:00.511Z debug: HEALTH: Server details: {"host":"pro2-win1.lab.ptarmiganlabs.net:4747","serverName":"sense1","serverDescription":"Central","logDbHost":"pro2-win1","userSessions":{"enable":true,"host":"pro2-win1.lab.ptarmiganlabs.net:4243","virtualProxies":[{"virtualProxy":"/"},{"virtualProxy":"/hdr"},{"virtualProxy":"/sales"},{"virtualProxy":"/sourcing"}]},"serverTags":null}
2023-05-05T11:31:00.511Z debug: SERVERTAGS: Complete list of tags for server sense1: {"host":"pro2-win1.lab.ptarmiganlabs.net","server_name":"sense1","server_description":"Central"}
2023-05-05T11:31:00.512Z debug: HEALTH: URL=https://pro2-win1.lab.ptarmiganlabs.net:4747/engine/healthcheck/
2023-05-05T11:31:00.518Z verbose: HEALTH: Getting stats for server: sense2
2023-05-05T11:31:00.518Z debug: HEALTH: Server details: {"host":"pro2-win2.lab.ptarmiganlabs.net:4747","serverName":"sense2","serverDescription":"Dev1","logDbHost":"pro2-win2","userSessions":{"enable":true,"host":"pro2-win2.lab.ptarmiganlabs.net:4243","virtualProxies":[{"virtualProxy":"/"},{"virtualProxy":"/hdr"}]},"serverTags":null}
2023-05-05T11:31:00.518Z debug: SERVERTAGS: Complete list of tags for server sense2: {"host":"pro2-win2.lab.ptarmiganlabs.net","server_name":"sense2","server_description":"Dev1"}
2023-05-05T11:31:00.518Z debug: HEALTH: URL=https://pro2-win2.lab.ptarmiganlabs.net:4747/engine/healthcheck/
2023-05-05T11:31:00.534Z verbose: HEALTH: Received ok response from pro2-win2.lab.ptarmiganlabs.net
2023-05-05T11:31:00.535Z debug: HEALTH: {"version":"12.1501.3","started":"20230412T032225.000+0200","mem":{"committed":3200.453125,"allocated":6741.58984375,"free":23620.484375},"cpu":{"total":0},"session":{"active":0,"total":0},"apps":{"active_docs":[],"loaded_docs":[],"in_memory_docs":[],"calls":40539,"selections":0},"users":{"active":0,"total":0},"cache":{"hits":0,"lookups":0,"added":0,"replaced":0,"bytes_added":0},"saturated":false}
2023-05-05T11:31:00.535Z debug: HEALTH: Calling HEALTH metrics MQTT posting method
2023-05-05T11:31:00.538Z debug: HEALTH: Calling HEALTH metrics Influxdb posting method
2023-05-05T11:31:00.539Z debug: HEALTH METRICS TO INFLUXDB: Health data: Tags sent to InfluxDB: {"host":"pro2-win2.lab.ptarmiganlabs.net","server_name":"sense2","server_description":"Dev1"}
2023-05-05T11:31:00.539Z debug: HEALTH METRICS TO INFLUXDB: Number of apps active: 0
2023-05-05T11:31:00.540Z debug: HEALTH METRICS TO INFLUXDB: Number of apps loaded: 0
2023-05-05T11:31:00.540Z debug: HEALTH METRICS TO INFLUXDB: Number of apps in memory: 0
2023-05-05T11:31:00.546Z verbose: HEALTH: Received ok response from pro2-win1.lab.ptarmiganlabs.net
2023-05-05T11:31:00.546Z debug: HEALTH: {"version":"12.1501.3","started":"20230411T223041.000+0200","mem":{"committed":771.5,"allocated":3688.96484375,"free":58014.28515625},"cpu":{"total":0},"session":{"active":0,"total":1},"apps":{"active_docs":[],"loaded_docs":["e377d9b5-231a-4795-800f-3649983647dd"],"in_memory_docs":["2933711d-6638-41d4-a2d2-6dd2d965208b","a3e0f5d2-000a-464f-998d-33d333b175d7","e377d9b5-231a-4795-800f-3649983647dd"],"calls":37087,"selections":0},"users":{"active":0,"total":1},"cache":{"hits":0,"lookups":0,"added":0,"replaced":0,"bytes_added":0},"saturated":false}
2023-05-05T11:31:00.546Z debug: HEALTH: Calling HEALTH metrics MQTT posting method
2023-05-05T11:31:00.548Z debug: HEALTH: Calling HEALTH metrics Influxdb posting method
2023-05-05T11:31:00.548Z debug: HEALTH METRICS TO INFLUXDB: Health data: Tags sent to InfluxDB: {"host":"pro2-win1.lab.ptarmiganlabs.net","server_name":"sense1","server_description":"Central"}
2023-05-05T11:31:00.548Z debug: HEALTH METRICS TO INFLUXDB: Number of apps active: 0
2023-05-05T11:31:00.548Z debug: HEALTH METRICS TO INFLUXDB: Number of apps loaded: 1
2023-05-05T11:31:00.548Z debug: HEALTH METRICS TO INFLUXDB: Number of apps in memory: 3
2023-05-05T11:31:01.013Z verbose: MEMORY USAGE INFLUXDB: Sent Butler SOS memory usage data to InfluxDB
2023-05-05T11:31:01.286Z verbose: HEALTH METRICS: Sent health data to Influxdb for server sense2
2023-05-05T11:31:01.287Z verbose: HEALTH METRICS: Sent health data to Influxdb for server sense1

If you look closer at this particular line, the app IDs of the in-memory-docs are there:

2023-05-05T11:31:00.546Z debug: HEALTH: {"version":"12.1501.3","started":"20230411T223041.000+0200","mem":{"committed":771.5,"allocated":3688.96484375,"free":58014.28515625},"cpu":{"total":0},"session":{"active":0,"total":1},"apps":{"active_docs":[],"loaded_docs":["e377d9b5-231a-4795-800f-3649983647dd"],"in_memory_docs":["2933711d-6638-41d4-a2d2-6dd2d965208b","a3e0f5d2-000a-464f-998d-33d333b175d7","e377d9b5-231a-4795-800f-3649983647dd"],"calls":37087,"selections":0},"users":{"active":0,"total":1},"cache":{"hits":0,"lookups":0,"added":0,"replaced":0,"bytes_added":0},"saturated":false}

[mountaindude]

... and because I am lazy I won't check in InfluxDB to verify that the data arrived there, but rather use Grafana for that purpose.
Easier when doing quick checks like this (and when building operations dashboards of course!).

Looks like the data made it to InfluxDB:

[mountaindude]

Now, what's causing your problem?
Really hard to say without knowing what your config file looks like.
Without that file I can't do much to help you, unfortunately.
Feel free to mask out sensitive info in that file if needed (IP addresses etc).

[mountaindude]

... but then again, I can of course share what my config looks like.

The interesting part is the influxdbConfig section:

  # Influx db config parameters
  influxdbConfig:
    enable: true
    # Items below are mandatory if influxdbConfig.enable=true
    hostIP: <ip or host name of server where InfluxDB is running)
    hostPort: 8086          # Optional. Default value=8086
    auth:
      enable: false             # Does influxdb instance require authentication (true/false)?
      # username: user_joe      # Username for Influxdb authentication. Mandatory if auth.enable=true
      # password: joesecret     # Password for Influxdb authentication. Mandatory if auth.enable=true
    dbName: senseops
    # Default retention policy that should be created in InfluxDB when Butler SOS creates a new database there. 
    # Any data older than retention policy threshold will be purged from InfluxDB.
    retentionPolicy:
      name: 10d
      duration: 10d                 # Possible duration units here: https://docs.influxdata.com/influxdb/v1.8/query_language/spec/#durations

    # Control whether certain metrics are stored in InfluxDB or not
    # Use with caution! Enabling activeDocs, loadedDocs or inMemoryDocs may result in lots of data sent to InfluxDB.
    includeFields:
      activeDocs: true              # Should data on what docs are active be stored in Influxdb (true/false)? 
      loadedDocs: true              # Should data on what docs are loaded be stored in Influxdb (true/false)?
      inMemoryDocs: true            # Should data on what docs are in memory be stored in Influxdb (true/false)?

[theidemannIf]

Thank you very much, this was the important hint, just enable it ...