Ansible operator v1.25.2 misses custom resource modify events

[venkataramanam]

Bug Report

What did you do?

We have multiple instances of a custom resource in our application. One among those we treat it as some kind of master instance. The others are treated as sort of logical instances. Whenever any updates are done to the master like upgrading it by specifying a field .spec.version, operator does all the necessary hard job of upgrading that instance. Finally at the very end of reconcile, the operator code updates .spec.version in all other logical instances. This act of updating is to synchronize the logical instances with the master. Once the master CR reconcile has been completed, Operator used to pick up change events in all logical instances and reconcile them as well. Please note that we have let the Operator to reconcile one at time (sequential and not parallel - --max-concurrent-reconciles=1). All of this used to work absolutely fine till now and when we upgraded to v1.25.2, we see the change events to logical events (the .spec.version updates) are not being getting reconciled. We dont know why.

This is the ansible-operator command line with its args...

ENTRYPOINT ["/tini", "--", "/usr/local/bin/ansible-operator", "run", \
    "--watches-file=./watches.yaml", \
    "--zap-log-level=error", \
    "--max-concurrent-reconciles=1", \
    "--ansible-verbosity=0", \
    "--reconcile-period=0s" \
    ]

The watches.yaml

- version: v1
  group: wos.cpd.ibm.com
  kind: WOService
  snakeCaseParameters: False
  playbook: reconcile.yaml
  watchDependentResources: False
  finalizer:
    name: wos.cpd.ibm.com/finalizer
    playbook: finalize.yaml

NOTE:

1. The reconcile-period is set to 0 to not reconcile the CRs periodically unless a change is made to a CR.
2. The logical instance CRs are not owned by the master instance CR, incase this has any bearing.

What did you expect to see?

We expected the change events in the logical instance CRs to be reconciled by the Operator.

What did you see instead? Under which circumstances?

We are seeing the change events in the logical instance CRs are not getting reconciled by the Operator. This used to work prior to us upgrading to v1.2.5.2 ...The last version was v1.25.0 where it used to work all good.

Environment

Operator type:

/language ansible

Kubernetes cluster type:

OpenShift
Server Version: 4.10.37
Kubernetes Version: v1.23.5+8471591
$ operator-sdk version

v1.25.2

$ go version (if language is Go)

$ kubectl version

Possible Solution

Additional context

[varshaprasad96]

@venkataramanam we don't have any particular changes that could affect the way ansible operator works in 1.25.2 from 1.25.0 (https://sdk.operatorframework.io/docs/upgrading-sdk-version/v1.25.2/). Could you add any error logs if they appear?
Is there a change in openshift version on which the operator is being run? Could you also check if there are any events associated with the logical instances which are different from the previous working release?

[venkataramanam]

@varshaprasad96

we don't have any particular changes that could affect the way ansible operator works in 1.25.2 from 1.25.0

Yes, I did see there aren't changes around this area.

Could you add any error logs if they appear?

Unfortunately, there aren't any error logs.

Is there a change in openshift version on which the operator is being run?

Yes, there is a change in the OpenShift version. Not sure if that was the reason for missing change events from being watched.

Could you also check if there are any events associated with the logical instances which are different from the previous working release?

No, absolutely no change in events. We only just update .spec.version in all logical instances as part of master CR reconcile at the very end.

Is there some sort of timeout beyond which even if there are change events, they would not get processed ?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.