helm operator upgrade is happening continuously without any trigger

[malli31]

I am seeing strange issue with Helm operator,
Installed helm operator setup and the tried installing
CR , after successful installation, Immediate upgrade is happening
and this process continuing , I am seeing below logs in operator pod continuously

{"level":"info","ts":1671986485.61458,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986545.2212965,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986606.602362,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986667.6448646,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986728.6824412,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986789.7362242,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}

[malli31]

Seeing this especially with helm-operator sdk 1.25.3

[varshaprasad96]

@malli31 Can you provide more details on what you mean by "immediate upgrade is happening". If possible, can you also point us towards your repository so that we can look at the code. Also, what is the version of operator-sdk you are using?

[malli31]

Dear @varshaprasad96 @laxmikantbpandhare

Thanks for responding,
helm operator sdk version we are using is 1.25.3
if we revert to 1.22.x working as expectred, anythign specific changed in 1.25.3
in operator pod log seeing Upgraded release info
{"level":"info","ts":1671986485.61458,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986545.2212965,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986606.602362,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}
{"level":"info","ts":1671986667.6448646,"logger":"helm.controller","msg":"Upgraded release","namespace":"caapm","name":"uma-monitor","apiVersion":"ca.broadcom.com/v1alpha1","kind":"UniversalMonitoringAgent","release":"uma-monitor","force":false}

please find git url where our metadata is present

https://github.com/redhat-openshift-ecosystem/certified-operators/tree/main/operators/uma-operator/2022.11.3

this is metadata that we used from our helm charts, let me know if you want ful helm-charts code?

[malli31]

If you want to try our operator in kubernetes you can install from
https://packages.broadcom.com/artifactory/apm-agents/uma/2022.11.0.13-1/uma-operator.tar.gz
untar uma-operator.tar.gz and go to uma-operator folder and run below commands,

kubectl create ns caapm
kubectl create -f setup -n caapm
kubectl create -f uma_cr.yaml -n caapm

and try to delete uma_cr jus installed
kubectl delete -f uma_cr.yaml -n caapm

its struck forever, and seeing reconisile error in uma-operator-*** pod,

also before running delete cr, we are continuously seeing Upgraded release info in uma-operator logs
and becuase of this i suspect sh.helm.release.v1.uma-monitor.v19 this hel secret versions are changin and this may be causing the reconsile errors?

If you want to insall in Openshift 4.11 jus go to operator hub and search for uma and install

[laxmikantbpandhare]

thank you @malli31 for a detailed explanation. I will take a look into it.

[malli31]

@laxmikantbpandhare any update on this, please we are not able to delete cr

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.