Skip to content

Cluster woker node 증설하기

Jump to bottom
jjuyoung edited this page Aug 5, 2023 · 1 revision

Cluster worker node 증설하기

현재 master node 하나에 taint 를 해지하여 pod 를 배포하는 상황이다. 조금 더 일반적인 환경을 구성하기 위하여 worker node 를 추가하여 테스트를 진행한다.

👩🏻‍💻 과정 정리

Master

master 노드에 taint 추가

kubectl taint nodes master node-role.kubernetes.io/control-plane=:NoSchedule

다음 명령어로 join 정보 확인하기

kubeadm token create --print-join-command

Worker

💡 헷갈리는 오라클의 NSG & Security List

NSG(네트워크 보안 그룹)와 Security List(보안 목록)은 모두 네트워크 접근 규칙을 정의하는 VCN 구성 요소이다.

  • Security List : 보안 목록의 보안 규칙은 대상 Subnet에 포함된 전체 VM에 일괄 적용된다.

  • NSG : Security List 보다 정교하게 규칙이 적용되는 대상 서버를 그룹으로 한정할 수 있다는 점이다.

kubernetes 환경 세팅하기

  • 자동화 스크립트

    docker_k8s_setting.sh

    #!/bin/bash

#------Swap Disable-----#
echo "Swap off"
sudo -i
swapoff -a
echo 0 > /proc/sys/vm/swappiness
sed -e '/swap/ s/^#*/#/' -i /etc/fstab

#------Install Docker-----#
echo "Install docker"
sudo apt update
sudo apt install -y docker.io
sudo systemctl start docker
sudo systemctl enable docker

cat << EOF | sudo tee –a /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo systemctl daemon-reload
sudo systemctl restart docker

#------Install k8s-----#
echo "Install kubernetes"
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
shell docker_k8s_setting.sh

master에 join하기(최종)

sudo kubeadm join 10.0.0.145:6443 --token 4ehb86.ll7m7sncl36xn0a7 --discovery-token-ca-cert-hash sha256:440f445a1e60ba3db69ba338a0861f4a0904d4e0628d02d2881c778007931e83

Kubeadm join failed : Failed to request cluster-info

I0724 17:33:08.977221    3834 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://10.0.0.145:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.0.0.145:6443: connect: no route to host

현재 마스터노드의 10.0.0.145:6443 로 ping은 잘 나가는 상황!

→ 근데 또 curl 은 안됨….

  • 상세 로그

    ubuntu@worker:~$ sudo kubeadm join 10.0.0.145:6443 --token vh3sbm.n8724z01747qlx2x --discovery-token-ca-cert-hash sha256:440f445a1e60ba3db69ba338a0861f4a0904d4e0628d02d2881c778007931 --v=5
I0724 17:45:27.023815    4813 join.go:412] [preflight] found NodeName empty; using OS hostname as NodeName
I0724 17:45:27.024021    4813 initconfiguration.go:117] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
[preflight] Running pre-flight checks
I0724 17:45:27.024114    4813 preflight.go:93] [preflight] Running general checks
I0724 17:45:27.024163    4813 checks.go:280] validating the existence of file /etc/kubernetes/kubelet.conf
I0724 17:45:27.024186    4813 checks.go:280] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0724 17:45:27.024203    4813 checks.go:104] validating the container runtime
I0724 17:45:27.057167    4813 checks.go:639] validating whether swap is enabled or not
I0724 17:45:27.057247    4813 checks.go:370] validating the presence of executable crictl
I0724 17:45:27.057295    4813 checks.go:370] validating the presence of executable conntrack
I0724 17:45:27.057330    4813 checks.go:370] validating the presence of executable ip
I0724 17:45:27.057358    4813 checks.go:370] validating the presence of executable iptables
I0724 17:45:27.057388    4813 checks.go:370] validating the presence of executable mount
I0724 17:45:27.057421    4813 checks.go:370] validating the presence of executable nsenter
I0724 17:45:27.057454    4813 checks.go:370] validating the presence of executable ebtables
I0724 17:45:27.057488    4813 checks.go:370] validating the presence of executable ethtool
I0724 17:45:27.057517    4813 checks.go:370] validating the presence of executable socat
I0724 17:45:27.057550    4813 checks.go:370] validating the presence of executable tc
I0724 17:45:27.057592    4813 checks.go:370] validating the presence of executable touch
I0724 17:45:27.057628    4813 checks.go:516] running all checks
I0724 17:45:27.073763    4813 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost
I0724 17:45:27.073979    4813 checks.go:605] validating kubelet version
I0724 17:45:27.142649    4813 checks.go:130] validating if the "kubelet" service is enabled and active
I0724 17:45:27.153971    4813 checks.go:203] validating availability of port 10250
I0724 17:45:27.154242    4813 checks.go:280] validating the existence of file /etc/kubernetes/pki/ca.crt
I0724 17:45:27.154270    4813 checks.go:430] validating if the connectivity type is via proxy or direct
I0724 17:45:27.154320    4813 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0724 17:45:27.154368    4813 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0724 17:45:27.154401    4813 join.go:529] [preflight] Discovering cluster-info
invalid hash "sha256:440f445a1e60ba3db69ba338a0861f4a0904d4e0628d02d2881c778007931", expected a 32 byte SHA-256 hash, found 30 bytes
k8s.io/kubernetes/cmd/kubeadm/app/util/pubkeypin.(*Set).Allow
        cmd/kubeadm/app/util/pubkeypin/pubkeypin.go:63
k8s.io/kubernetes/cmd/kubeadm/app/discovery/token.retrieveValidatedConfigInfo
        cmd/kubeadm/app/discovery/token/token.go:66
k8s.io/kubernetes/cmd/kubeadm/app/discovery/token.RetrieveValidatedConfigInfo
        cmd/kubeadm/app/discovery/token/token.go:53
k8s.io/kubernetes/cmd/kubeadm/app/discovery.DiscoverValidatedKubeConfig
        cmd/kubeadm/app/discovery/discovery.go:83
k8s.io/kubernetes/cmd/kubeadm/app/discovery.For
        cmd/kubeadm/app/discovery/discovery.go:43
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).TLSBootstrapCfg
        cmd/kubeadm/app/cmd/join.go:530
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).InitCfg
        cmd/kubeadm/app/cmd/join.go:540
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runPreflight
        cmd/kubeadm/app/cmd/phases/join/preflight.go:98
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
        cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598
invalid discovery token CA certificate hash
k8s.io/kubernetes/cmd/kubeadm/app/discovery/token.retrieveValidatedConfigInfo
        cmd/kubeadm/app/discovery/token/token.go:67
k8s.io/kubernetes/cmd/kubeadm/app/discovery/token.RetrieveValidatedConfigInfo
        cmd/kubeadm/app/discovery/token/token.go:53
k8s.io/kubernetes/cmd/kubeadm/app/discovery.DiscoverValidatedKubeConfig
        cmd/kubeadm/app/discovery/discovery.go:83
k8s.io/kubernetes/cmd/kubeadm/app/discovery.For
        cmd/kubeadm/app/discovery/discovery.go:43
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).TLSBootstrapCfg
        cmd/kubeadm/app/cmd/join.go:530
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).InitCfg
        cmd/kubeadm/app/cmd/join.go:540
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runPreflight
        cmd/kubeadm/app/cmd/phases/join/preflight.go:98
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
        cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598
couldn't validate the identity of the API Server
k8s.io/kubernetes/cmd/kubeadm/app/discovery.For
        cmd/kubeadm/app/discovery/discovery.go:45
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).TLSBootstrapCfg
        cmd/kubeadm/app/cmd/join.go:530
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).InitCfg
        cmd/kubeadm/app/cmd/join.go:540
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runPreflight
        cmd/kubeadm/app/cmd/phases/join/preflight.go:98
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
        cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598
error execution phase preflight
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:260
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
        cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

→ GPT 피셜 : 이 오류는 워커 노드에서 입력한 discovery-token-ca-cert-hash 값이 잘못되었기 때문에 발생하는 것입니다. 유효한 32바이트의 SHA-256 해시가 아닌 30바이트의 해시를 제공했기 때문에 오류가 발생하였습니다.

sudo kubeadm join --token 4ehb86.ll7m7sncl36xn0a7 10.0.0.145:6443 --discovery-token-ca-cert-hash sha256:440f445a1e60ba3db69ba338a0861f4a0904d4e0628d02d2881c778007931e83
💡 `--apiserver-advertise-address=10.0.0.145` 옵션을 넣어줘야 api-server를 잘 찾게 된다.

굉장히 킹 받는 부분

sudo kubeadm join 10.0.0.145:6443 --token 4ehb86.ll7m7sncl36xn0a7 --discovery-token-ca-cert-hash sha256:440f445a1e60ba3db69ba338a0861f4a0904d4e0628d02d2881c778007931e83 **--apiserver-advertise-address=10.0.0.145** -v=5

join 성공

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

worker 노드에 키 파일, 인증서 전달하기

  1. key 파일 복붙해서 생성 ssh-oci.key 
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAuSK9c65Zuu4z4ODDw5vUiIGGNUIcswp2z91xR2OzV/vI4atu
hOzZx21G2NHqvX77AUavfDBL9Y4g4acD9exQcS/Sxd8+3ihmO+gPjG6LuTP37X+Z
wJDpuV3HFNetBXkgJBNIfaYdkpZ0AXWxa5i3RL5KPBK/zbdBef3cdjEhRBj+nuMG
OxEaHRIP+CkYS17gJg4T7tBnwr7QqNiWjsfiO6YlPexczv/zQpP76mKSaEc0T9Y9
qANiQhoqK+3Dp8H77BJHYWxaTgfM8vSgZO74YDmHkX2G75DhibQKDT3+dPFsmzCi
Pmzopucze1H7L26dOb7+95b3hw1Tfy7cTL3pqwIDAQABAoIBAATQCE5QUgj2vvRa
ykdrikANUem+XrkMe4RWOR3YeEpAJ/ucGgd1kZrcn5LIC+ZUdFmEyRjmSe/HspBo
1Y3hYq9fsxkI/2rg3h3FRFh9/lEhrqO9qewtgVcJmmV4F9KNrAUBxVHb/4s7VUw9
Rg/uw0xKiTQzANwiysMEJXbsOnVdv4Kj5P+5dwZDQQ6CcvgZeKRCW4iN7Uu5KUYJ
DE2swVotkem9ChakHdyRBaOlpgizcCIdU5mqTj6IHYK5yZVofyZlzfUMYo3U5bB3
0Q/P6AwRryKOu0+YzDHZ5U12rqtizdyzKAvls2RsfRR8U+WmTCvQaRS44NGsNPCk
t0ILA0UCgYEA/svPTHGT0hjWzNBy/oUxVVLRW4YLyuLtQlagllFI3Qx2qLS4l351
sHfhe0qJ87+OMCdCS9rK7OXs2QPHzs9mpOe68gPHCJIllPi1QliFGIo4XiIvHQec
jk+Csug5naAPw0tytQZ4qCNjbAOqU5oqsWT/uu4F7habG/FMLQqqLR0CgYEAugKs
DaZi3If+nTX5mwnp+5u2xIeW25Z+v71tphRFfxTsC9J204UUuxhnBDM0D8+J3kiU
dawXGimD9epa0zsyzV4UEXTzNMktxyiX0OhdfODq8gk97PbIQNccWuwiA61+hgUR
SZIQ40w5VRjiCjm8E0UXUOs6yeE007mOrdsoX2cCgYByIIJTAFCw+7VC+l93x9so
F7d04l7n2qthWoIFspULhiikOoawPNGkYJ4b5nBSQMwEn6EfbvoUEIz9S6c06EtW
tAFVpAGDwkqZJqmx9QpwyoCjEXa+7wjd6ZlmX4RnKSnGVUYk9agXz/oZiK7DUbur
x8yQQ0tesLt7UaLITg2ZtQKBgQCW6C4P0Oopwkcl4jFDv/JNR5IqtVoFgn4ILrVz
ZMTRCfdKzA/NIQ2lkHGbLbNSFGzkonuXT+taNU4yx1OyRTe891NOqlPZglGaXutz
9+BIvOy/OhI8WiYyvJHrjeE2nJ2mOxxAVLlQvnlQ+aQA6cAU3hvBcWW7fPLvxn3j
vIZvhwKBgEgCqT9hcgYUdlNDgIwgxwE03cf7+IFrZgTBGVhcrtmU5m4LF/OTLPBD
ADjoyKhYV6eZ14C/o5roBVyysrWeJj67nzXtuLtzPFASV2QgverLUbr5m8CkMDh1
WWMncgDpSlttb3GmBd9brY1KcSIzuisgKnF2fUsSgev41G9JFSB9
-----END RSA PRIVATE KEY-----
sudo chmod 400 ssh-oci.key
  1. scp 명령어로 master에서 worker로 전달(worker1, worker2 각각에 대해 실행)
scp -i ssh-oci.key -r oci-key/ [email protected]:oci-key/
sudo scp -i ssh-oci.key -r /etc/kubernetes/pki/etcd/ [email protected]:/etc/kubernetes/pki/etcd/

디렉토리가 한 단계 더 들어가 있어서, mv 해줌

(worker2는 지우언니가 worker2 키를 가지고 있어서 아직 못함)

worker에 맞게, volume path 변경

음? 그랬더니 이제는 snapshot이 안찍힘ㅎㅎㅎㅎㅎㅎㅎㅎㅎㅎㅎ

우라질 이놈의 no route to host

ETCD_ENDPOINT=10.0.0.145
INFO: 2023/07/26 18:44:22 parsed scheme: ""
INFO: 2023/07/26 18:44:22 scheme "" not registered, fallback to default scheme
INFO: 2023/07/26 18:44:22 ccResolverWrapper: sending update to cc: {[{10.0.0.145:2379 0  <nil>}] <nil>}
INFO: 2023/07/26 18:44:22 balancerWrapper: got update addr from Notify: [{10.0.0.145:2379 <nil>}]
WARNING: 2023/07/26 18:44:22 grpc: addrConn.createTransport failed to connect to {10.0.0.145:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.0.145:2379: connect: no route to host". Reconnecting...
WARNING: 2023/07/26 18:44:22 grpc: addrConn.createTransport failed to connect to {10.0.0.145:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.0.145:2379: connect: no route to host". Reconnecting...
WARNING: 2023/07/26 18:44:23 grpc: addrConn.createTransport failed to connect to {10.0.0.145:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.0.145:2379: connect: no route to host". Reconnecting...
WARNING: 2023/07/26 18:44:23 grpc: addrConn.createTransport failed to connect to {10.0.0.145:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.0.145:2379: connect: no route to host". Reconnecting...
Error:  dial tcp 10.0.0.145:2379: connect: no route to host

(근데 일단 워커노드에서 kubectl 명령이 안되는데(kubeconfig가 없어서) 아마 그것 때문일지도..? 원래 joing하면 kubeconfig도 자동으로 복사된다는데 우라질…)

근데 살펴보니 master의 ca.crt와 worker의 ca.crt가 달라서, 옮겨서 바꿔줌

(gpt 피셜: etcdctl을 worker 노드에서 실행할 때는 worker 노드의 ca.crt를 옵션으로 주어야 합니다)

그러나 실.패….

🤕 결과

worker node 에서는 etcd 와 관련된 작업을 할 수 없다. 따라서 master node 에서 작업을 하고 테스트하기로 결정 !!

References

[Port 6443 connection refused when setting up kubernetes](https://stackoverflow.com/questions/70571312/port-6443-connection-refused-when-setting-up-kubernetes)

→ kubeadm init 시의 --apiserver-advertise-address 옵션이 필요하다는 정보

[kubeadm join](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/)

CRI v1 runtime API is not implemented 이슈도 있었는데,, 이걸로 해결,,

Clone this wiki locally