EnvTest intercluster communication

[sempex]

Hello Everyone,
I am currently using EnvTest to write tests for my Operator, which I developed using controller-runtime and kubebuilder. The Operator uses a client.Client to create a Secret on a different Kubernetes cluster. This setup works fine with standard Kubernetes clusters. However, I wanted to test this system by setting up two EnvTest clusters.
Unfortunately, I encountered the following error:

"controller": "cluster", "controllerGroup": "[cluster.x-k8s.io](http://cluster.x-k8s.io/)", "controllerKind": "Cluster", "Cluster": {"name":"e1-k8s-pfnet-lab-x","namespace":"kube-cluster-management"}, "namespace": "kube-cluster-management", "name": "e1-k8s-pfnet-lab-x", "reconcileID": "1f2010a7-e073-4757-95df-adeace777737", "error": "Put \"https://127.0.0.1:35383/apis/cluster.x-k8s.io/v1beta1/namespaces/kube-cluster-management/clusters/e1-k8s-pfnet-lab-x\": dial tcp 127.0.0.1:35383: connect: connection refused"}

This error occurs when the Operator running on the first EnvTest cluster tries to create the Secret on the second EnvTest cluster. Is inter-cluster communication not possible with EnvTest?
Thank you for your help!

[camilamacedo86]

Hi @sempex

I think the controller-runtime maintainers would be the best ones to help on it.
It seems for me that ENVTEST is not designed to handle multi-cluster setups, but I will defer it to them. (@vincepri @sbueringer @alvaroaleman)

[sempex]

That would be awesome thank you!

[sbueringer]

Not sure I understand what you mean with "inter-cluster communication". envtest is not a real cluster, there is nothing running "in" that cluster. So communication between that cluster and another cluster doesn't make sense, I think

[camilamacedo86]

@sempex

Yes, your question is not clear.
See that when you use ENVTEST to test your controllers you will use the binaries which has the k8s api and etc.
More info: https://book.kubebuilder.io/reference/envtest

Which are:

    ├── etcd
    ├── kube-apiserver
    └── kubectl

So, you will not have a "inter-cluster communication".

However, it also allows you configure to test against an specific cluster by USE_EXISTING_CLUSTER .
When you use this option the tests will be executed against the cluster informed instead of use the binaries.
Is that the option that you are using?
Could you please clarify what are you trying to achieve?

[camilamacedo86]

Closing this one since no interactions were done anymore and the required info above was not provided.