Policy Reports Extension (PRExt)

Inspired by https://github.com/cmurphy/hns-list

This project is a Kubernetes API Extension to store Policy reports and cluster policy reports in an external database instead of etcd.

Why move policy reports outside of etcd?: kyverno/KDP#51

It is desirable to move reports out of etcd for several reasons:

Why leave etcd:

• The etcd database currently has a maximum size of 8GB. Reports tend to be relatively large objects, but even with very small reports, the etcd capacity can be easily reached in larger clusters with many report producers.
• Under heavy report activity (e.g. cluster churn, scanning, analytical processes, etc.), the volume of data being written and retrieved by etcd requires the API server to buffer large amounts of data. This compounds existing cluster issues and can cascade into complete API unavailability.
• CAP guarantees are not required for reports, which, at present, are understood to be ephemeral data which will be re-created if deleted.
• Philosophically, report data is analytical in nature and should not be stored in the transactional database.
• These are also the arguments for removing Kubernetes Events from the primary etcd as well. This issue has not yet been fully resolved.

Expected benefits of alternative storage:

• Alleviation of the etcd + API server load and capacity limitations.
• Common report consumer workflows can be more efficient.
  • Report consumers are often analytical in nature and/or operate on aggregate data. The API is not designed to efficiently handle, for example, a query for all reports where containing a vulnerability with a CVSS severity of 8.0 or above. To perform such a query, a report consumer must retrieve and parse all of the reports. Retrieving a large volume of reports, especially with multiple simultaneous consumers, leads to the performance issues described previously.
  • With reports stored in, for instance, a relational database, report consumers could instead query the underlying database directly, using more robust query syntax.
  • This would improve the implementation of, or even replace the need for, certain exporters, and enable new reporting use cases.

This extension allows us store policy reports outside of etcd.

Build

Build the kubectl-prext plugin and install it in your GOPATH:

make cli

Build the docker image for server:

make server

Install

1. Install cert-manager

2. Apply the manifest:

kubectl apply -f manifest/manifest.yaml

Usage

Using Raw API Queries

Create a new policy report:

kubectl create --raw /apis/prext.demo/v1alpha1/namespaces/default/policyreports -f config/testdata/testpolicy.json

Output

{
    "kind": "PolicyReport",
    "apiVersion": "prext.demo/v1alpha1",
    "metadata": {
      "name": "test",
      "namespace": "default",
      "uid": "8f611e66-e398-401d-9624-fd9ec87a72f8",
      "creationTimestamp": null,
      "ownerReferences": [
        {
          "apiVersion": "v1",
          "kind": "Pod",
...
...

Get the all policy reports in a namespace:

kubectl get --raw /apis/prext.demo/v1alpha1/namespaces/{{NAMESPACE}}/policyreports | jq --args ".[].metadata.name"      

Output:

{
    "kind": "PolicyReportList",
    "apiVersion": "prext.demo/v1alpha1",
    "metadata": {
      "resourceVersion": "31622"
    },
    "items": [
      {
        "kind": "PolicyReport",
        "apiVersion": "prext.demo/v1alpha1",
        "metadata": {
          "name": "image-scan-pod-identity-webhook-6d5f85488d-2vdjn",
...
...

View a policy report:

kubectl get --raw /apis/prext.demo/v1alpha1/namespaces/default/policyreports/test1

Output:

{
    "kind": "PolicyReport",
    "apiVersion": "prext.demo/v1alpha1",
    "metadata": {
      "name": "image-scan-pod-identity-webhook-6d5f85488d-2vdjn",
      "namespace": "default",
      "uid": "87428580-1dbf-4f2c-8320-069c9db00367",
      "resourceVersion": "125248938",
      "generation": 19,
      "creationTimestamp": "2023-10-18T15:33:08Z",
      "labels": {
        "kubernetes.io/createdBy": "vulnerability-operator"
...
...

Delete a policy report:

kubectl delete --raw /apis/prext.demo/v1alpha1/namespaces/default/policyreports/test 

View a cluster policy report:

kubectl get --raw /apis/prext.demo/v1alpha1/clusterpolicyreports/test1 

Output:

{
    "kind": "ClusterPolicyReport",
    "apiVersion": "prext.demo/v1alpha1",
    "metadata": {
      "name": "kube-bench",
      "uid": "ed0d888c-fab5-4597-98f4-d93bbd2a28d6",
      "resourceVersion": "130199899",
      "generation": 3,
      "creationTimestamp": "2023-10-22T00:00:14Z",
      "managedFields": [
        {
          "manager": "policyreport",
          "operation": "Update",
          "apiVersion": "prext.demo/v1alpha1",
          "time": "2023-10-22T00:00:14Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:results": {},
            "f:summary": {
              ".": {},
              "f:error": {},
              "f:fail": {},
              "f:pass": {},
              "f:skip": {},
              "f:warn": {}
            }
          }
        }
      ]
    },
    "summary": {
      "pass": 14,
      "fail": 0,
      "warn": 2,
...
...

Get all cluster policy reports:

kubectl get --raw /apis/prext.demo/v1alpha1/clusterpolicyreports

Output:

{
    "kind": "ClusterPolicyReportList",
    "apiVersion": "prext.demo/v1alpha1",
    "metadata": {
      "resourceVersion": "31778"
    },
    "items": [
      {
        "kind": "ClusterPolicyReport",
        "apiVersion": "prext.demo/v1alpha1",
        "metadata": {
          "name": "kube-bench",
          "uid": "ed0d888c-fab5-4597-98f4-d93bbd2a28d6",
          "resourceVersion": "130199899",
          "generation": 3,
          "creationTimestamp": "2023-10-22T00:00:14Z",
          "managedFields": [
            {
              "manager": "policyreport",
              "operation": "Update",
              "apiVersion": "prext.demo/v1alpha1",
...
...

Using kubectl plugin

Create a new policy report:

kubectl prext create -f config/testdata/testpolicy.json

PolicyReport 'test' in namespace default successfully created.

Get yaml of a policy report

kubectl prext get polr test -o yaml

apiVersion: ""
kind: ""
metadata:
    creationTimestamp: null
    name: test
    namespace: default
    ownerReferences:
        - apiVersion: v1
          kind: Pod
          name: test-amutate
          uid: 6c737f35-260c-4e4f-8516-61a125c9fb78
    uid: 8f611e66-e398-401d-9624-fd9ec87a72f8
results:
    - message: validation rule 'call-aws-signer-extension' passed.
      policy: validate-images
      result: pass
      rule: call-aws-signer-extension
      scored: true
      source: kyverno
...
...

Get policy reports in a namespace

kubectl prext get polr -n test-ns

Policy reports in namespace: test-ns
NAME
image-scan-cp-provider-aws-ee6bece46dbb-85654fc6b4-xpbn5
image-scan-kyverno-notation-aws-59bb56b89c-wxpcm

Patch a policy report

kubectl prext apply -f config/testdata/testpolicy.json

PolicyReport 'test' in namespace default successfully configured.

Delete a policy report

kubectl prext delete polr test

Successfully deleted  polr :  test