Calico node failed to start due to no such file or directory name="/usr/local/bin/felix-plugins/felix-api-proxy"

[dylan2intel]

Environment

• Calico/VPP version: both v3.23 and v3.24
• Kubernetes version: 1.24
• Deployment type: bare-metal...
• Network configuration: Calico /VXLAN enabled

Issue description
2022-12-23 07:54:43.193 [WARNING][32] tunnel-ip-allocator/param_types.go 294: Path lookup failed error=exec: "/usr/local/bin/felix-plugins/felix-api-proxy": stat /usr/local/bin/felix-plugins/felix-api-proxy: no such file or directory name="/usr/local/bin/felix-plugins/felix-api-proxy"
2022-12-23 07:54:43.193 [ERROR][32] tunnel-ip-allocator/param_types.go 299: Executable missing
2022-12-23 07:54:43.193 [ERROR][32] tunnel-ip-allocator/config_params.go 559: Invalid (required) config value. error=Failed to parse config parameter DataplaneDriver; value "/usr/local/bin/felix-plugins/felix-api-proxy": missing file source=environment variable
2022-12-23 07:54:43.193 [PANIC][32] tunnel-ip-allocator/allocateip.go 776: Failed to parse Felix environments error=Failed to parse config parameter DataplaneDriver; value "/usr/local/bin/felix-plugins/felix-api-proxy": missing file
panic: (*logrus.Entry) 0xc00041f650

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0010ffe18, 0x0, {0xc0008fe480, 0x22})
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:259 +0x24f
github.com/sirupsen/logrus.(*Entry).Log(0xc00041f5e0, 0x0, {0xc0010ffe18, 0x2a423c0, 0xc000489810})
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:293 +0x4f
github.com/sirupsen/logrus.(*Entry).Panic(...)
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:331
github.com/projectcalico/calico/node/pkg/allocateip.loadFelixEnvConfig()
/go/src/github.com/projectcalico/calico/node/pkg/allocateip/allocateip.go:776 +0xa5
github.com/projectcalico/calico/node/pkg/allocateip.Run(0xc000914420)
/go/src/github.com/projectcalico/calico/node/pkg/allocateip/allocateip.go:68 +0x7b
main.main()
/go/src/github.com/projectcalico/calico/node/cmd/calico-node/main.go:169 +0x518
Calico node failed to start

To Reproduce
Steps to reproduce the behavior:

• kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.23/manifests/tigera-operator.yaml
• Download and edit file with custom cidr https://projectcalico.docs.tigera.io/archive/v3.23/manifests/custom-resources.yaml
• kubectl apply -f custom-resources.yaml

Expected behavior
Calico node successfully to start

Additional context
Add any other useful information about the problem or your deployment here.

[sknat]

Hi @dylan2intel
/usr/local/bin/felix-plugins/ should be mounted in the felix pod by the operator and the felix-api-proxy binary copied in this folder when the calico/vpp agent pod starts.

On startup the calico/vpp agent will wait for the calico/vpp vpp manager to start. What probably happens is something is preventing your VPP to start, the agent waits indefinitely, and never creates the proxy binary

Do you see anything wrong in the output of kubectl -n calico-vpp-dataplane logs calico-vpp-node-XXXX -c vpp ?

[chairwa]

problem reason might be that the following cmd was not executed:
kubectl create -f calico-vpp.yaml