In MeshHTTPRoute with MeshMultizoneService XDS config is invalid

[lahabana]

What happened?

on version: 0.0.0-preview.vacb7899a5

With the following MeshHTTPRoute:

apiVersion: kuma.io/v1alpha1
kind: MeshHTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kuma.io/v1alpha1","kind":"MeshHTTPRoute","metadata":{"annotations":{},"labels":{"kuma.io/mesh":"default","kuma.io/origin":"zone"},"name":"demo-app","namespace":"kong-mesh-system"},"spec":{"targetRef":{"kind":"MeshGateway","name":"demo-app"},"to":[{"rules":[{"default":{"backendRefs":[{"kind":"MeshService","name":"demo-app_kuma-demo_svc_5000"}]},"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}],"targetRef":{"kind":"Mesh"}}]}}
    kuma.io/display-name: demo-app
  creationTimestamp: "2024-09-18T12:42:36Z"
  generation: 3
  labels:
    k8s.kuma.io/namespace: kong-mesh-system
    kuma.io/env: kubernetes
    kuma.io/mesh: default
    kuma.io/origin: zone
    kuma.io/policy-role: system
    kuma.io/zone: west
  name: demo-app
  namespace: kong-mesh-system
  ownerReferences:
  - apiVersion: kuma.io/v1alpha1
    kind: Mesh
    name: default
    uid: c048cf62-1adb-4047-b6f4-f89e02aee21b
  resourceVersion: "61213"
  uid: 8fac6858-66ea-4b56-9f56-c24fc4ca9a57
spec:
  targetRef:
    kind: MeshGateway
    name: demo-app
  to:
  - rules:
    - default:
        backendRefs:
        - kind: MeshMultiZoneService
          labels:
            kuma.io/display-name: demo-app-everywhere
          weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
    targetRef:
      kind: Mesh

And meshMZService:

type: MeshMultiZoneService
mesh: default
name: demo-app-everywhere
spec:
  selector:
    meshService:
      matchLabels:
        kuma.io/display-name: demo-app
  ports:
  - name: '5000'
    port: 5000

2024-09-18T13:34:27.348Z	INFO	xds.nack-backoff	config was previously rejected by Envoy. Applying backoff before resending it	{"backoff": "5s", "nodeID": "default.demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo", "reason": "At least one of name or cluster_header need to be specified"}
2024-09-18T13:34:32.350Z	INFO	xds.status-tracker	config rejected	{"proxyName": "demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo", "mesh": "default", "streamID": 3, "type": "RouteConfiguration", "resourceVersion": "95f3b4d2-a59c-4ed6-8829-4e056d93bd6a"}
2024-09-18T13:34:32.350Z	INFO	xds.nack-backoff	config was previously rejected by Envoy. Applying backoff before resending it	{"backoff": "5s", "nodeID": "default.demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo", "reason": "At least one of name or cluster_header need to be specified"}

It's possible that my route is invalid but it should behave differently anyway

[lahabana]

My zone meshmultizoneservice:

➜  ~ kubectl get meshmultizoneservice -A -oyaml
apiVersion: v1
items:
- apiVersion: kuma.io/v1alpha1
  kind: MeshMultiZoneService
  metadata:
    annotations:
      kuma.io/display-name: demo-app-everywhere
    creationTimestamp: "2024-09-18T12:59:26Z"
    generation: 230
    labels:
      kuma.io/mesh: default
      kuma.io/origin: global
    name: demo-app-everywhere-cdzbd69dxz92wf94
    namespace: kong-mesh-system
    resourceVersion: "68393"
    uid: 14099451-3ddb-4973-b1b3-9accd5b945c1
  spec:
    ports:
    - appProtocol: tcp
      name: "5000"
      port: 5000
      targetPort: 0
    selector:
      meshService:
        matchLabels:
          kuma.io/display-name: demo-app
  status:
    addresses:
    - hostname: demo-app-everywhere.mzsvc.mesh.local
      hostnameGeneratorRef:
        coreName: synced-mesh-multi-zone-service-4244v574644557w2.kong-mesh-system
      origin: HostnameGenerator
    hostnameGenerators:
    - conditions:
      - message: ""
        reason: Generated
        status: "True"
        type: Generated
      hostnameGeneratorRef:
        coreName: synced-mesh-multi-zone-service-4244v574644557w2.kong-mesh-system
    meshServices:
    - mesh: default
      name: demo-app
      namespace: kuma-demo
      zone: east
    - mesh: default
      name: demo-app
      namespace: kuma-demo
      zone: west
    vips:
    - ip: 243.0.0.0
kind: List
metadata:
  resourceVersion: ""

So the label I have doesn't exist

[lahabana]

Tried using:

        backendRefs:
        - kind: MeshMultiZoneService
          tags:
            kuma.io/display-name: demo-app-everywhere
          weight: 1

no success either.

with:

        backendRefs:
        - kind: MeshMultiZoneService
          name: demo-app-everywhere-<hash>
          weight: 1

The error becomes:

2024-09-18T13:44:08.744Z	INFO	meshmultizoneservice.status-updater	updating matched mesh services	{"meshmultizoneservice": "demo-app-everywhere-cdzbd69dxz92wf94.kong-mesh-system", "matchedMeshServices": [{"name":"demo-app","namespace":"kuma-demo","zone":"west","mesh":"default"},{"name":"demo-app","namespace":"kuma-demo","zone":"east","mesh":"default"}]}
2024-09-18T13:44:09.454Z	INFO	plugin.runtime.gateway	skipping backendRef	{"backendRefError": "json: unsupported type: func() *model.LegacyBackendRef"}
2024-09-18T13:44:09.454Z	INFO	plugin.runtime.gateway	skipping backendRef	{"backendRefError": "json: unsupported type: func() *model.LegacyBackendRef"}

[lahabana]

Yet another error if I use a new MeshService:

    - default:
        backendRefs:
        - kind: MeshService
          labels:
            kuma.io/disploy-name: demo-app
          weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /

(hitting the same issue when fixing the label typo)

2024-09-18T13:50:00.454Z	ERROR	xds.server.dataplane-sync-watchdog	OnTick() failed	{"dataplaneKey": {"Mesh":"default","Name":"demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo"}, "error": "could not reconcile: failed to generate a snapshot: imports[0]{name=\"gateway-proxy\"}: generator.generator failed: could not apply policy plugin meshhttproute: failed to generate clusters for dataplane {\"default\" \"demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo\"}: cluster name is undefined", "errorVerbose": "imports[0]{name=\"gateway-proxy\"}: generator.generator failed: could not apply policy plugin meshhttproute: failed to generate clusters for dataplane {\"default\" \"demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo\"}: cluster name is undefined\nfailed to generate a snapshot\ngithub.com/kumahq/kuma/pkg/xds/server/v3.(*reconciler).Reconcile\n\tgithub.com/kumahq/[email protected]/pkg/xds/server/v3/reconcile.go:57\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).syncDataplane\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:166\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).Sync\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:80\ngithub.com/kumahq/kuma/pkg/xds/sync.(*dataplaneWatchdogFactory).New.func2\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog_factory.go:44\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).onTick\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:54\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).Start\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:22\nruntime.goexit\n\truntime/asm_amd64.s:1700\ncould not reconcile\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).syncDataplane\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:168\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).Sync\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:80\ngithub.com/kumahq/kuma/pkg/xds/sync.(*dataplaneWatchdogFactory).New.func2\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog_factory.go:44\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).onTick\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:54\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).Start\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:22\nruntime.goexit\n\truntime/asm_amd64.s:1700"}

[lahabana]

For MeshService it was caused by missing port. This works:

    - default:
        backendRefs:
        - kind: MeshService
          labels:
            kuma.io/display-name: demo-app
          port: 5000
          weight: 1

So it's a validation issue

[lahabana]

Other validation issue I use:

        backendRefs:
        - kind: MeshMultiZoneService
          name: demo-app-everywhere
          port: 5000
          weight: 1

I get in the CP logs:

2024-09-18T14:33:34.454Z	INFO	plugin.runtime.gateway	skipping backendRef	{"backendRefError": "json: unsupported type: func() *model.LegacyBackendRef"}
2024-09-18T14:33:34.454Z	INFO	plugin.runtime.gateway	skipping backendRef	{"backendRefError": "json: unsupported type: func() *model.LegacyBackendRef"}

[lahabana]

Also when using a label selector that doesn't match anything I get:

2024-09-18T14:35:40.455Z	ERROR	xds.server.dataplane-sync-watchdog	OnTick() failed	{"dataplaneKey": {"Mesh":"default","Name":"demo-app-gateway-7b9f64bc74-ww9wz.kuma-demo"}, "error": "could not reconcile: invalid resource \"demo-app:HTTP:80:*\": invalid RouteConfiguration.VirtualHosts[0]: embedded message failed validation | caused by: invalid VirtualHost.Routes[0]: embedded message failed validation | caused by: invalid Route.Action: value is required; invalid VirtualHost.Routes[1]: embedded message failed validation | caused by: invalid Route.Action: value is required", "errorVerbose": "invalid RouteConfiguration.VirtualHosts[0]: embedded message failed validation | caused by: invalid VirtualHost.Routes[0]: embedded message failed validation | caused by: invalid Route.Action: value is required; invalid VirtualHost.Routes[1]: embedded message failed validation | caused by: invalid Route.Action: value is required\ninvalid resource \"demo-app:HTTP:80:*\"\ngithub.com/kumahq/kuma/pkg/xds/server/v3.(*reconciler).Reconcile\n\tgithub.com/kumahq/[email protected]/pkg/xds/server/v3/reconcile.go:85\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).syncDataplane\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:166\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).Sync\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:80\ngithub.com/kumahq/kuma/pkg/xds/sync.(*dataplaneWatchdogFactory).New.func2\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog_factory.go:44\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).onTick\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:54\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).Start\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:22\nruntime.goexit\n\truntime/asm_amd64.s:1700\ncould not reconcile\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).syncDataplane\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:168\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).Sync\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog.go:80\ngithub.com/kumahq/kuma/pkg/xds/sync.(*dataplaneWatchdogFactory).New.func2\n\tgithub.com/kumahq/[email protected]/pkg/xds/sync/dataplane_watchdog_factory.go:44\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).onTick\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:54\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).Start\n\tgithub.com/kumahq/[email protected]/pkg/util/watchdog/watchdog.go:22\nruntime.goexit\n\truntime/asm_amd64.s:1700"}

[michaelbeaumont]

Have #11458 to fix the log message

[michaelbeaumont]

@lahabana I opened #11465 and I think then all problems in this issue have been handled