FKNMS InfluxDB container crashes every few days.

[dotis]

I triggered DAG runs for FKNMS and FGBNMS dashes and now FK is not working, although it did work yesterday. FGB seems fine. I wonder if this is the same docker issue?

[7yl4r]

Looks like it failed for 2 days then went back to working. The error message does look related to #53 (InfluxDB outage):

[2023-07-31 13:41:56,351] {subprocess.py:78} INFO - requests.exceptions.ConnectionError: HTTPConnectionPool(host='35.211.75.134', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES&db=fwc_coral_disease (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f196406e820>: Failed to establish a new connection: [Errno 111] Connection refused'))

Closing as resolved by #53

[dotis]

Need to re-open this. DAGs not running and getting bad gateway error w/no time series data for FK. FGB board seems fine.

[7yl4r]

This "bad gateway" error means that the gateway to InfluxDB isn't working, and that is almost always because the influxDB container has died.

tylarmurray@fknms-dashboard-04:~$ docker container ls -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED        STATUS                      PORTS                                                 NAMES
[...]
1353c06aedc5   influxdb:1.8                               "/entrypoint.sh infl…"   6 weeks ago    Exited (137) 5 days ago                                                           influxdb
[...]

Same as before in #53 there is again nothing in the logs.
I can restart this easily whenever it goes down, but we need to identify why the container is crashing so I can stop it.
There may be logs within the container somewhere that I can dig into.
I will try that next time; for now I am pouring all spare energy into the POSE proposal.