Instructions

Installation of a "hello-world" project

For testing we are going to use this "hello-world" app - https://gist.githubusercontent.com/vitobotta/6e73f724c5b94355ec21b9eee6f626f1/raw/3036d4c4283a08ab82b99fffea8df3dded1d1f78/deployment.yaml

Install kubectl on your computer: https://kubernetes.io/docs/tasks/tools/#kubectl
Install Helm on your computer: https://helm.sh/docs/intro/install/
Install hetzner-k3s on your computer: Installation
Create file hetzner-k3s_cluster_config.yaml with the config below (this is a config for High Available (HA) cluster with 3 master nodes + 3 worker nodes. You can use 1+1 for testing):

hetzner_token: ...
cluster_name: hello-world
kubeconfig_path: "./kubeconfig"  # or /cluster/kubeconfig if you are going to use Docker
k3s_version: v1.23.3+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.ssh/id_rsa.pub"
    private_key_path: "~/.ssh/id_rsa"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0

masters_pool:
  instance_type: cpx21
  instance_count: 3
  location: nbg1

worker_node_pools:
- name: small
  instance_type: cpx21
  instance_count: 4
  location: hel1
- name: big
  instance_type: cpx31
  instance_count: 2
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 3

Refer to the full config example in - Creating a cluster for details on all the settings that can be customized

Create cluster: hetzner-k3s create --config hetzner-k3s_cluster_config.yaml
hetzner-k3s automatically creates a kubeconfigfile for the cluster in the directory of your computer where you run the tool, so you can either copy the kubeconfig file to ~/.kube/config if it's the only cluster, or run export KUBECONFIG=./kubeconfig in the same directory to access the cluster. Then you can interact with your cluster via kubectl installed in the 1st step.

TIP: If you don't want to run kubectl apply ... every time, you can store all configs in some folders and then run kubectl apply -f /path/to/configs/ -R.

Create file: touch ingress-nginx-annotations.yaml
Add annotations to the file: nano ingress-nginx-annotations.yaml

# INSTALLATION
# 1. Install Helm: https://helm.sh/docs/intro/install/
# 2. Add ingress-nginx help repo: helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# 3. Update information of available charts locally from chart repositories: helm repo update
# 4. Install ingress-nginx:
# helm upgrade --install \
# ingress-nginx ingress-nginx/ingress-nginx \
# --set controller.ingressClassResource.default=true \ # remove this line if you don't want Nginx to become the default Ingress Controller
# -f ./ingress-nginx-annotations.yaml \
# --namespace ingress-nginx \
# --create-namespace

# LIST of all ANNOTATIONS: https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/internal/annotation/load_balancer.go

controller:
  kind: DaemonSet
  service:
    annotations:
      # Germany:
      # - nbg1 (Nuremberg)
      # - fsn1 (Falkensteing)
      # Finland:
      # - hel1 (Helsinki)
      # USA:
      # - ash (Ashburn, Virginia)
      # Without this the load balancer won't be provisioned and will stay in "pending" state.
      # The state you can check via "kubectl get svc -n ingress-nginx"
      load-balancer.hetzner.cloud/location: nbg1

      # Name of load balancer. This name you will see in your Hetzner's cloud console (site) at the "Your project -> Load Balancers" page
      # NOTE: This is NOT the load balancer that the tool creates automatically for clusters with multiple masters (HA configuration). You need
      # to specify a different name here so it will create a separate load balancer for ingress Nginx.
      load-balancer.hetzner.cloud/name: WORKERS_LOAD_BALANCER_NAME

      # Ensures that the communication between the load balancer and the cluster nodes happens through the private network
      load-balancer.hetzner.cloud/use-private-ip: "true"

      # [ START: If you care about seeing the actual IP of the client then use these two annotations ]
      # - "uses-proxyprotocol" enables the proxy protocol on the load balancers so that ingress controller and
      # applications can "see" the real IP address of the client.
      # - "hostname" is needed just if you use cert-manager (LetsEncrypt SSL certificates). You need to use it in order
      # to fix fails http01 challenges of "cert-manager" (https://cert-manager.io/docs/).
      # Here (https://github.com/compumike/hairpin-proxy) you can find a description of this problem.
      # To be short: the easiest fix provided by some providers (including Hetzner) is to configure the load balancer so
      # that it uses a hostname instead of an IP.
      load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'

      # 1. "yourDomain.com" must be configured in the DNS correctly to point to the Nginx load balancer,
      # otherwise the provision of certificates won't work;
      # 2. if you use a few domains, specify any one.
      load-balancer.hetzner.cloud/hostname: yourDomain.com
      # [ END: If you care about seeing the actual IP of the client then use these two annotations ]

      load-balancer.hetzner.cloud/http-redirect-https: 'false'

Add ingress-nginx Helm repo: helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
Update information of available charts locally from chart repositories: helm repo update
Install ingress-nginx:

helm upgrade --install \
ingress-nginx ingress-nginx/ingress-nginx \
--set controller.ingressClassResource.default=true \
-f ~/.kube/ingress-nginx-annotations.yaml \
--namespace ingress-nginx \
--create-namespace

--set controller.ingressClassResource.default=true will configure this to be the default Ingress Class for your cluster. Without a default, you must specify an Ingress Class for every Ingress object you deploy, which is often difficult when deploying Helm Charts. If you do not set a default Ingress Class nor specify one on the Ingress resource, Nginx will serve a 404 Not Found page, as it did not "pick up" the Ingress resource.

TIP: Just in case you need to delete it: helm uninstall ingress-nginx -n ingress-nginx. Be careful, this will delete current Hetzner's load balancer and as a result when you install a new ingress controller, a new Hetzner's load balancer possibly will be created with a new public IP address.

In a few minutes check that the "EXTERNAL-IP" column has IP instead of "pending": kubectl get svc -n ingress-nginx
load-balancer.hetzner.cloud/uses-proxyprotocol: "true" annotation requires use-proxy-protocol: "true" for ingress-nginx, so let's create file: touch ingress-nginx-configmap.yaml
Add content to just created file: nano ingress-nginx-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  # Do not change name - this is the name required by Nginx Ingress Controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  use-proxy-protocol: "true"

Apply config map: kubectl apply -f ./ingress-nginx-configmap.yaml
Open your Hetzner's cloud console (site), "Your project -> Load Balancers" and find PUBLIC IP in front of the name you used with "load-balancer.hetzner.cloud/name: WORKERS_LOAD_BALANCER_NAME" annotation. Copy/Remember this IP.
Download hello-world app: curl https://gist.githubusercontent.com/vitobotta/6e73f724c5b94355ec21b9eee6f626f1/raw/3036d4c4283a08ab82b99fffea8df3dded1d1f78/deployment.yaml --output hello-world.yaml
Edit the file (add annotation + add Hetzner's Load Balancer IP Address) and set the hostname:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:                             # <<<--- Add annotation
    kubernetes.io/ingress.class: nginx     # <<<--- Add annotation
spec:
  rules:
  - host: hello-world.IP_FROM_STEP_13.nip.io # <<<--- ADD IP FROM THE STEP 16.
  ....

Install hello-world app: kubectl apply -f hello-world.yaml
Check http://hello-world.IP_FROM_STEP_13.nip.io You should see the RANCHER Hello world! page. "host.IP_FROM_STEP_13.nip.io" (the key part is ".nip.io") is just a quick way to test things without configuring DNS (a query to a hostname ending in nip.io simply returns the IP address it finds in the hostname itself). Also, if you enabled proxy-protocol as shown above, you should find your own current public IP address in the X-Forwarded-For header - i.e. the application can "see" it.
In order to connect yourDomain.com, you need to:

assign IP address from the step 13 to your domain in DNS panel of your domain registrar
change - host: hello-world.IP_FROM_STEP_13.nip.io to - host: yourDomain.com;
kubectl apply -f hello-world.yaml
wait until DNS records are updated.

If you need LetsEncrypt

Add LetsEncrypt Helm repo: helm repo add jetstack https://charts.jetstack.io
Update information of available charts locally from chart repositories: helm repo update
Install LetsEncrypt certificates issuer:

helm upgrade --install \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true \
cert-manager jetstack/cert-manager

Create file lets-encrypt.yaml with content:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: YOUR@EMAIL.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx

Apply file: kubectl apply -f ./lets-encrypt.yaml
Edit hello-world.yaml and add settings for TLS encryption:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"    # <<<--- Add annotation
    kubernetes.io/tls-acme: "true"                        # <<<--- Add annotation
spec:
  rules:
  - host: yourDomain.com  # <<<---- Your real domain
  tls: # <<<---- Add this block
  - hosts:
    - yourDomain.com
    secretName: yourDomain.com-tls # <<<--- Add reference to secret

  ....

TIP: if you chose not to configure Nginx as the default Ingress Class, you must add the kubernetes.io/ingress.class: nginx annotation.

Apply changes: kubectl apply -f ./hello-world.yaml

FAQs

1. Is it possible to use for example MetalLB instead of Hetzner's LB?

There is a way to use MetalLB with floating IPs in Hetzner Cloud but I don't recommend it. The setup with standard load balancers is much simpler and load balancers are not that much more expensive than floating IPs so IMO there's no point using MetalLB.

2. How to create and push docker images to a repository and how to allow kubernetes to work with this image (gitlab example)?

On a computer which creates an image:

docker login registry.gitlab.com
docker build -t registry.gitlab.com/COMPANY_NAME/REPO_NAME:IMAGE_NAME -f /some/path/to/Dockerfile .
docker push registry.gitlab.com/COMPANY_NAME/REPO_NAME:IMAGE_NAME

On a computer which runs kubernetes:

generate secret to access images: kubectl create secret docker-registry gitlabcreds --docker-server=https://registry.gitlab.com --docker-username=MYUSER --docker-password=MYPWD --docker-email=MYEMAIL -n NAMESPACE_OF_YOUR_APP -o yaml > docker-secret.yaml
apply secret: kubectl apply -f docker-secret.yaml -n NAMESPACE_OF_YOUR_APP

3. How to check how much resources nodes/pods use?

Install metrics-server https://github.com/kubernetes-sigs/metrics-server
Then use either kubectl top nodes or kubectl top pods -A

4. What is Ingress?

There are 2 types of "ingress" -> Ingress Controller and Ingress Resources. To simplify everything, in the case of Nginx...

Ingress Controller is Nginx itself (this is kind: Ingress), Ingress Resources are services (ie. kind: Service).
Ingress Controller has different annotations (rules). You can use them inside kind: Ingress as a result such rules become "global" and inside kind: Service as a result such rules become "local" (service-specific).
Ingress Controller consists of a Pod and a Service. The Pod runs the Controller, which constantly polls the /ingresses endpoint on the API server of your cluster for updates to available Ingress Resources.

5. How to make autoscaling configure automatically IP routes to use a NAT server for new nodes?

You need to have a NAT server - as explained in this Hetzner community tutorial.
Use post_create_commands (multiple lines commands don't seem to be supported at the moment):

additional_packages:
  - ifupdown
post_create_commands:
  - apt update
  - apt upgrade -y
  - apt autoremove -y
  - ip route add default via 10.0.0.1  # Adapt this to your gateway IP

Useful commands

kubectl get service [serviceName] -A or -n [nameSpace]
kubectl get ingress [ingressName] -A or -n [nameSpace]
kubectl get pod [podName] -A or -n [nameSpace]
kubectl get all -A
kubectl get events -A
helm ls -A
helm uninstall [name] -n [nameSpace]
kubectl -n ingress-nginx get svc
kubectl describe ingress -A
kubectl describe svc -n ingress-nginx
kubectl delete configmap nginx-config -n ingress-nginx
kubectl rollout restart deployment -n NAMESPACE_OF_YOUR_APP
kubectl get all -A` does not include "ingress", as a result you need to use `kubectl get ing -A

Useful links

Cheat Sheet - https://kubernetes.io/docs/reference/kubectl/cheatsheet/ A visual guide on troubleshooting Kubernetes deployments - https://learnk8s.io/troubleshooting-deployments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setting up a cluster.md

Setting up a cluster.md

Instructions

Installation of a "hello-world" project

If you need LetsEncrypt

FAQs

1. Is it possible to use for example MetalLB instead of Hetzner's LB?

2. How to create and push docker images to a repository and how to allow kubernetes to work with this image (gitlab example)?

3. How to check how much resources nodes/pods use?

4. What is Ingress?

5. How to make autoscaling configure automatically IP routes to use a NAT server for new nodes?

Useful commands

Useful links

Files

Setting up a cluster.md

Latest commit

History

Setting up a cluster.md

File metadata and controls

Instructions

Installation of a "hello-world" project

If you need LetsEncrypt

FAQs

1. Is it possible to use for example MetalLB instead of Hetzner's LB?

2. How to create and push docker images to a repository and how to allow kubernetes to work with this image (gitlab example)?

3. How to check how much resources nodes/pods use?

4. What is Ingress?

5. How to make autoscaling configure automatically IP routes to use a NAT server for new nodes?

Useful commands

Useful links