Helm chart deployment Mattermost Enterprise Edition(8.1.13) onto Openshift 4.14.24 - minio permission issues & mysqlha pods unhealthy

[marlowsw]

Summary
Helm deployment of mattermost enterprise edition(8.1.13) on OpenShift 4.14.24 as a testing environment for production. Also, tested on local minikube v1.33.1 which fails as well. However this does work on Openshift 4.12.56 without issue.(deployment on 4.12.56 isn’t an option)

Steps to reproduce
Clone repo: mattermost-helm/charts/mattermost-enterprise-edition at master · mattermost/mattermost-helm · GitHub
Follow configuration steps in the readme.
Deploy with oc install -f config.yaml -n namespace

Expected behavior
mattermost deployed and ready to login. Works on OpenShift 4.12.56

Observed behavior
minio bucket is seeing a permissions issue:
mysqlha pods deploy in unhealthy state

Connecting to Minio server: http://mattermost-test-minio:9000
mc: Unable to save new mc config. mkdir /etc/minio: permission denied.
"Failed attempts: 1"
mysqlha-1 is in crashloopbackOff:

++ hostname

• [[ mattermost-test-mysqlha-1 =~ -([0-9]+)$ ]]
• ordinal=1
• [[ 1 -eq 0 ]]
• [[ -d /var/lib/mysql/mysql ]]
• ncat --recv-only mattermost-test-mysqlha-0.mattermost-test-mysqlha 3307
• xbstream -x -C /var/lib/mysql
  Ncat: Connection refused.
• xtrabackup --prepare --user=repl --password=RQYPVsuuIwk7 --target-dir=/var/lib/mysql
  xtrabackup version 2.4.4 based on MySQL server 5.7.13 Linux (x86_64) (revision id: df58cf2)
  xtrabackup: cd to /var/lib/mysql
  xtrabackup: Error: cannot open ./xtrabackup_checkpoints
  xtrabackup: error: xtrabackup_read_metadata()
  xtrabackup: This target seems not to have correct metadata...
  InnoDB: Number of pools: 1
  InnoDB: Operating system error number 2 in a file operation.
  InnoDB: The error means the system cannot find the path specified.
  xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
  InnoDB: Operating system error number 2 in a file operation.
  InnoDB: The error means the system cannot find the path specified.
  xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
  xtrabackup: Error: xtrabackup_init_temp_log() failed.
  mattermost-test platform-test-atlassian 1 2024-06-10 12:25:16.692405 -0400 EDT failed mattermost-enterprise-edition-2.6.34 8.1.13

64m (x3 over 64m) Warning Unhealthy Pod/mattermost-test-mysqlha-0 Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of dce1a6de204bd043c1371f761d510da903ac303484882b23c837bb7368588cb0 is running failed: container process not found
61m (x2 over 61m) Warning BadConfig Ingress/mattermost-test-mattermost-enterprise-edition Could not determine issuer for ingress due to bad annotations: failed to determine issuer name to be used for ingress resource
60m (x2 over 60m) Warning BadConfig Ingress/mattermost-test-mattermost-enterprise-edition Could not determine issuer for ingress due to bad annotations: failed to determine issuer name to be used for ingress resource
59m (x4 over 60m) Warning Unhealthy Pod/mattermost-test-mattermost-enterprise-edition-99756dc59-4jgxv Readiness probe failed: Get "http://10.35.8.46:8065/api/v4/system/ping": dial tcp 10.35.8.46:8065: connect: connection refused
59m (x5 over 60m) Warning Unhealthy Pod/mattermost-test-mattermost-enterprise-edition-99756dc59-cvrzs Readiness probe failed: Get "http://10.34.9.14:8065/api/v4/system/ping": dial tcp 10.34.9.14:8065: connect: connection refused
50m (x20 over 58m) Warning BackOff Pod/mattermost-test-minio-make-bucket-job-7hs2x Back-off restarting failed container minio-mc in pod mattermost-test-minio-make-bucket-job-7hs2x_platform-test-atlassian(b68bcb33-3800-4f57-84b5-ac4cc4a47aa6)
30m (x140 over 60m) Warning BackOff Pod/mattermost-test-mysqlha-1 Back-off restarting failed container clone-mysql in pod mattermost-test-mysqlha-1_platform-test-atlassian(28789cb7-cf76-4e67-a47c-7cc63adf192f)
26m (x3 over 26m) Warning Unhealthy Pod/mattermost-test-mysqlha-0 Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of 5d880e60479454dd87743fc64f5bb056c64a6be5eda49dbd403bbd95129d3430 is running failed: container process not found
21m Warning BadConfig Ingress/mattermost-test-mattermost-enterprise-edition Could not determine issuer for ingress due to bad annotations: failed to determine issuer name to be used for ingress resource
20m (x3 over 21m) Warning Unhealthy Pod/mattermost-test-mysqlha-0 Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of a3e6f135cda0f630572caf44705b16adda8ab54e7354c0713954877b365aa1f6 is running failed: container process not found
20m Warning BadConfig Ingress/mattermost-test-mattermost-enterprise-edition Could not determine issuer for ingress due to bad annotations: failed to determine issuer name to be used for ingress resource
19m (x4 over 19m) Warning Unhealthy Pod/mattermost-test-mattermost-enterprise-edition-99756dc59-wbl9x Readiness probe failed: Get "http://10.35.8.53:8065/api/v4/system/ping": dial tcp 10.35.8.53:8065: connect: connection refused
10m (x21 over 18m) Warning BackOff Pod/mattermost-test-minio-make-bucket-job-qd8h7 Back-off restarting failed container minio-mc in pod mattermost-test-minio-make-bucket-job-qd8h7_platform-test-atlassian(7f2fc6cb-272d-491d-b89c-beeee8eedd3c)
4m51s (x72 over 19m) Warning BackOff Pod/mattermost-test-mysqlha-1 Back-off restarting failed container clone-mysql in pod mattermost-test-mysqlha-1_platform-test-atlassian(844c1071-2d55-4fbb-b51f-5bc0a29bf932)

I do end up with a running instance however the deployment is in a “failed” state and the pods aren’t all healthy.
I’d appreciate any direction I can get on solving this.