Why do we need Kyverno ?

Problem

Imagine a scenario where we have a Node with 5GB Memory and accidentally deployed a pod with exactly 5GB Memory. In that case no other pod would be deployed and all of them would be in pending state

Solution

So Kyverno is an Advanced Admission Controller where it evaluates, based on rules whether to deploy certain resoucres or not. In our case if we enforce a Kyverno policy that sets resources limit to not cross say 2GB, if we deployed a pod with 5GB Memory it would reject our request

High Level Design

To explain briefly, we write our kyverno policy and push it github repo. Since ArgoCD is configured to watch for changes in the Repo and sync it with the cluster, ArgoCD will deploy the necessary changes to our Namespace.

The Heart of Kyverno

Kyverno is a policy engine designed for Kubernetes

A Kyverno policy is a collection of rules. Each rule consists of a match declaration, an optional exclude declaration, and one of a validate, mutate, generate, or verifyImages declaration. Each rule can contain only a single validate, mutate, generate, or verifyImages child declaration.

Policies can be defined as cluster-wide resources (using the kind ClusterPolicy) or namespaced resources (using the kind Policy.) As expected, namespaced policies will only apply to resources within the namespace in which they are defined while cluster-wide policies are applied to matching resources across all namespaces. Otherwise, there is no difference between the two types.

Install Kyverno Using Manifest Files

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.8.5/install.yaml

We created two Kyverno Policies

1. Enforce pod-requests-and-limits --> This policy makes sure that each Deployment is created with resource limits

2. Enforce memory-limit --> This policy ensures that memory limts in a pod don't cross a certain value

   Example YAML file where memory specified is greater than the limit

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod-exceed-limits
  name: pod-exceed-limits
spec:
  containers:
  - image: nginx
    name: pod-exceed-limits
    resources: 
      requests:
        memory: "4Gi" # Should not exceed 2GB
        cpu: "250m"
      limits:
        memory: "5Gi"
        cpu: "500m"
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Using Kyverno, we can

1. Generate --> For example, Create a default network policy whenever a namespace is created.

2. Validate --> For example, Enforce CPU and Memory Resource Limits

3. Mutate --> For example, Add Sidecar Container for Logging to Deployments

4. Verify Images --> For example, Verify if the Images used in the pod resources are properly signed and verified images.

The required Kyverno policy is written and pushed to Git repo. ArgoCD deploys it onto the Kubernetes Cluster

ArgoCD

Components of ArgoCD

Repo Server --> This is used to watch the state of the version control system like github

Application Server --> This watches the state of Kubernetes Cluster

API Server --> It is useful for interacting with ArgoCD

Dex --> Provides authentication

Redis --> Used for caching

ArgoCD Insallation Using Helm

 helm repo add argo https://argoproj.github.io/argo-helm

 helm install argo argo/argo-cd

Testing

Case1. Auto-Sync

Our Cluster has no pods deployed in the default namespace

We upload a deployment yaml file in our git repo

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
spec:
  selector:
    matchLabels:
      app: nginx-app
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
      - name: ng
        image: nginx
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

# Seperate files in github
---
apiVersion: v1
kind: Service
metadata:
  name: nginxpp-service
spec:
  selector:
    app: nginx-app 
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080

We now proceed to build ArgoCD

1. Source - Our github link and the path is our folder name where yaml files are stored

2. Destination - Where our K8S cluster is running

3. Note - Selfheal and pruning are disabled by default

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-test
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/satya19977/argo-kyverno-testing
    targetRevision: HEAD
    path: argo-kyverno
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
     selfHeal: true
     prune: true  

ArgoCD uses git as the single source of truth and synced our cluster with Git repo

As Evident, we have all the resources defined in our Git

Case2. Self-Healing

If this is enabled, any change made directly in the cluster isn't enforced.

Let's increase the number of replicas directly in the cluster and observer what happens

Even though we manually made change to our cluster since selfheal is enabled, argocd will not push those changes

Case3. Enable Prune

Since prune is enabled any change in the Git will also reflect in our cluster.

Changing the name of the deployment from nginx-app to deployment-nginx has caused the previous deployment to be deleted and the deployment with the new name was created

Cae4. Let's deploy a pod with no limits and resources and see how kyverno reacts.

This is the yaml file in our git

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-nginx
spec:
  selector:
    matchLabels:
      app: nginx-app
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
      - name: ng
        image: nginx
        ports:
        - containerPort: 8080

We see here that the pod has been deployed because in our kyverno policy we set validatefailureaction to audit and not enforce. So that's why our pod has been deployed but the logs give us a warning to enforce resource limits in our pod