Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig-win.conf' because it does not exist #377

Open
pbdiazam opened this issue Sep 23, 2024 · 0 comments
Open

Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig-win.conf' because it does not exist #377

pbdiazam opened this issue Sep 23, 2024 · 0 comments

Comments

@pbdiazam
Copy link

pbdiazam commented Sep 23, 2024 •
edited
Loading

Describe the bug
When I deploy kube-proxy which matches my k8s version, it enters into CrashLoopBackoff:
image

I can see kube-proxy-windows starts running and after a little time it fails and restarts because it cannot find kubeconfig file path, and these are the logs I see:

kubectl logs -f kube-proxy-windows-6l7s4 -n kube-system

WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Running kub-proxy service.
Waiting for HNS network Calico to be created...
HNS network Calico found.
kubeproxy version Kubernetes v1.30.4
Write files so the kubeconfig points to correct locations


    Directory: C:\var\lib


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         9/23/2024   7:53 AM                kube-proxy
Get-Content : Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig.conf' because it does not exist.
At C:\hpc\kube-proxy\start.ps1:56 char:3
+ ((Get-Content -path $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/kube-p ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\hpc\var\lib\...kubeconfig.conf:String) [Get-Content], ItemNotFoundEx
   ception
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand

cp : Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig-win.conf' because it does not exist.
At C:\hpc\kube-proxy\start.ps1:57 char:1
+ cp $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/kube-proxy/kubeconfig-w ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\hpc\var\lib\...config-win.conf:String) [Copy-Item], ItemNotFoundExce
   ption
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.CopyItemCommand

Requires 2019 with KB4580390 (Oct 2020)
Detected VXLAN network, waiting for Calico host endpoint to be created...
Host endpoint found.
Enabling feature gates: WinDSR=true WinOverlay=true.
Start to run C:\hpc\/kube-proxy/kube-proxy.exe --hostname-override=ip-100-90-2-213.ec2.internal --v=4 --proxy-mode=kernelspace --kubeconfig=C:\hpc\/var/lib/kube-proxy/kubeconfig-win.conf --enable-dsr=true --source-vip=172.16.181.2 --feature-gates=WinDSR=true,WinOverlay=true
I0923 07:54:51.665208    2664 flags.go:64] FLAG: --bind-address="0.0.0.0"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --bind-address-hard-fail="false"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --cleanup="false"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --cluster-cidr=""
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --config=""
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --config-sync-period="15m0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-max-per-core="32768"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-min="131072"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-tcp-be-liberal="false"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-udp-timeout="0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-udp-timeout-stream="0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --detect-local-mode=""
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --enable-dsr="true"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --feature-gates="WinDSR=true,WinOverlay=true"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --forward-healthcheck-vip="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --healthz-bind-address="0.0.0.0:10256"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --healthz-port="10256"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --help="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --hostname-override="ip-100-90-2-213.ec2.internal"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --init-only="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-localhost-nodeports="true"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-masquerade-bit="14"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-min-sync-period="1s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-sync-period="30s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-exclude-cidrs="[]"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-min-sync-period="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-scheduler=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-strict-arp="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-sync-period="30s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-tcp-timeout="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-tcpfin-timeout="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-udp-timeout="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kube-api-burst="10"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kube-api-qps="5"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kubeconfig="C:\\hpc\\/var/lib/kube-proxy/kubeconfig-win.conf"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-flush-frequency="5s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-json-info-buffer-size="0"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-json-split-stream="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-text-info-buffer-size="0"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-text-split-stream="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --logging-format="text"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --masquerade-all="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --master=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --metrics-port="10249"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --network-name=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --nodeport-addresses="[]"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --oom-score-adj="-999"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --pod-bridge-interface=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --pod-interface-name-prefix=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --profiling="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --proxy-mode="kernelspace"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --proxy-port-range=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --root-hnsendpoint-name="cbr0"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --show-hidden-metrics-for-version=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --source-vip="172.16.181.2"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --v="4"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --version="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --vmodule=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --windows-service="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --write-config-to=""
I0923 07:54:51.737319    2664 feature_gate.go:254] feature gates: {map[WinDSR:true WinOverlay:true]}
E0923 07:54:51.738038    2664 server.go:558] "Error running ProxyServer" err="CreateFile C:\\hpc\\/var/lib/kube-proxy/kubeconfig-win.conf: The system cannot find the file specified."
E0923 07:54:51.738038    2664 run.go:74] "command failed" err="CreateFile C:\\hpc\\/var/lib/kube-proxy/kubeconfig-win.conf: The system cannot find the file specified."

To Reproduce

EKS Setup with Linux and Calico Networking
eksctl create cluster --name cluster --region us-east-1 --vpc-private-subnets=subnet-1,subnet-2 --node-private-networking --without-nodegroup

kubectl delete daemonset -n kube-system aws-node

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/tigera-operator.yaml

kubectl create -f - <<EOF
kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
EOF

cat <<EOF  | kubectl apply -f -
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}
EOF

eksctl create nodegroup   --cluster cluster  --name linux-nodegroup   --subnet-ids subnet-3,subnet-4 --node-type t3.medium   --nodes 2   --nodes-min 0   --nodes-max 4   --managed=false   --region us-east-1 --node-private-networking

WINDOWS
aws iam list-attached-role-policies --role-name eksClusterRole
{
    "AttachedPolicies": [
        {
            "PolicyName": "AmazonEKSClusterPolicy",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
        },
        {
            "PolicyName": "AmazonEKSVPCResourceController",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
        }
    ]
}

aws iam attach-role-policy \
  --role-name eksClusterRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController

vi vpc-resource-controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  enable-windows-ipam: "true"

kubectl apply -f vpc-resource-controller-configmap.yaml

eksctl create nodegroup --cluster=cluster --name windows-nodegroup --node-ami-family=WindowsServer2022FullContainer --subnet-ids subnet-3,subnet-4 --node-type m5.xlarge --nodes 1   --nodes-min 1   --nodes-max 4   --managed=false   --region us-east-1 --node-private-networking

curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm.yaml

sed -i.bak -e 's|<ARN of instance role (not instance profile)>|<windows_nodegrouo_instance_role_arn>|' aws-auth-cm.yaml

kubectl apply -f aws-auth-cm.yaml

kubectl edit configmap aws-auth -n kube-system
Add the following to the config map:
- eks:kube-proxy-windows

kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'

kubectl patch installation default --type=merge -p '{"spec": {"calicoNetwork": {"bgp": "Disabled"}}}'

APISERVER_ADDR=<eks_server_endpoint>
APISERVER_PORT=443

kubectl apply -f - << EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: tigera-operator
data:
  KUBERNETES_SERVICE_HOST: "${APISERVER_ADDR}"
  KUBERNETES_SERVICE_PORT: "${APISERVER_PORT}"
EOF

kubectl patch installation default --type merge --patch='{"spec": {"serviceCIDRs": ["172.20.0.0/16"], "calicoNetwork": {"windowsDataplane": "HNS"}}}'

curl -L  https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml | sed "s/KUBE_PROXY_VERSION/v1.30.4/g" | kubectl apply -f -

Expected behavior
I would expect kube-proxy to work without any additional configuration and to be able to find the kubeconfig.

Kubernetes (please complete the following information):

  • Windows Server version: Windows Server 2022 Datacenter (AMI:WindowsServer2022FullContainer) - 10.0.20348.2700
  • Kubernetes Version: v1.30.4-eks-a737599
  • CNI: Calico v3.28.1

Additional context
These are the nodes I currently have in my cluster with the containerd version:
image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pbdiazam