Skip to content

Commit

Permalink
feat(helm chart): Add metrics-server hardening options
Browse files Browse the repository at this point in the history
  • Loading branch information
@mkilchhofer
mkilchhofer committed Jul 14, 2023
1 parent 796fc0f commit 0254b50
Show file tree
Hide file tree
Showing 8 changed files with 339 additions and 4 deletions.
130 changes: 130 additions & 0 deletions charts/metrics-server/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,133 @@ The following table lists the configurable parameters of the _Metrics Server_ ch
| `topologySpreadConstraints` | Pod Topology Spread Constraints. | `[]` |
| `deploymentAnnotations` | Annotations to add to the deployment. | `{}` |
| `schedulerName` | scheduler to set to the deployment. | `""` |
| `tls.type` | TLS option to use. Either use `metrics-server` for self-signed certificates, `secret`, `existingSecret`, `cert-manager` or `helm` | `"metrics-server"` |
| `tls.certManager.clusterDomain` | Kubernetes cluster domain. Used to configure Subject Alt Names for the certificate | `"cluster.local"` |
| `tls.certManager.addInjectorAnnotations` | Automatically add the cert-manager.io/inject-ca-from annotation to the APIService resource. | `true` |
| `tls.certManager.existingIssuer.enabled` | Use an existing cert-manager issuer | `false` |
| `tls.certManager.existingIssuer.kind` | Kind of the existing cert-manager issuer | `"Issuer"` |
| `tls.certManager.existingIssuer.name` | Name of the existing cert-manager issuer | `"my-issuer"` |
| `tls.certManager.duration` | Set the requested duration (i.e. lifetime) of the Certificate. | `""` |
| `tls.certManager.renewBefore` | How long before the currently issued certificate’s expiry cert-manager should renew the certificate. | `""` |
| `tls.certManager.annotations` | Add extra annotations to the Certificate resource | `{}` |
| `tls.certManager.labels` | Add extra labels to the Certificate resource | `{}` |
| `tls.existingSecret.name` | Name of the existing Secret to use for TLS | `""` |
| `tls.secret.annotations` | Add extra annotations to the Secret resource | `{}` |
| `tls.secret.labels` | Add extra labels to the Secret resource | `{}` |
| `tls.secret.crt` | The server certificate to use for metrics-server. Use PEM format | `""` |
| `tls.secret.key` | The private key of the certificate to use for metrics-server. Use PEM format. | `""` |
| `extraObjects` | List of extra manifests to deploy. Will be passed through `tpl` to support templating | `[]` |

## Hardening metrics-server

By default, metrics-server is using a self-signed certificate which is generated during startup. The APIservice is registered with `.spec.insecureSkipTLSVerify` set to `true` as you can see here:

```yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
#..
insecureSkipTLSVerify: true # <-- see here
service:
name: metrics-server
#..
```

To harden metrics-server, you have these 4 options described in the following section.

### Option 1: Let helm generate a self-signed certificate

This option is probably the easiest solution for you. We delegate helm the generation process for self-signed certificates.
As helm generates them during deploy time, helm can also inject the `apiService.caBundle` for you.
**The only disadvantage of using this method is that it is not GitOps friendly** (e.g. Argo CD). If you are using one of these
GitOps tools with drift detection, it will always detect changes. However if you are deploying the helm chart via Terraform
for example (or maybe even Flux), this method is perfectly fine.

To use this method, please setup your values file like this:

```yaml
apiService:
insecureSkipTLSVerify: false
tls:
type: helm
```
### Option 2: Use cert-manager
> **Requirement:** cert-manager needs to be installed before you install metrics-server
To use this method, please setup your values file like this:
```yaml
apiService:
insecureSkipTLSVerify: false
tls:
type: cert-manager
```
There are other optional parameters, if you want to customize the behavior of the certificate even more.
### Option 3: Provide certificate data
You can use an arbitrary PKI solution and generate a certificate for metrics-server. You need the following data in PEM format:
- The server certificate which is issued with 3 Subject Alt Names:
- `metrics-server.<namespace>`
- `metrics-server.<namespace>.svc`
- `metrics-server.<namespace>.svc.cluster.local`
- The server key file
- The CA certificate *(optional, you can also provide the server certificate to `apiService.caBundle`)*

To use this method, please setup your values file like this:

```yaml
apiService:
insecureSkipTLSVerify: false
caBundle: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls:
type: secret
secret:
crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
```

### Option 4: Use existing Secret

This option allows you to reuse an existing Secret. This Secrets can have an arbitrary origin, e.g.

- Created via kubectl / Terraform / etc.
- Synced from a secrets management solution like AWS SecretsManager, HashiCorp Vault, etc.

You still need to pass the CA certificate to ensure proper configuration of the `APIservice` resource,
but the sensitive information (the private key) can be read via Secret. Same as in the previous option "2",
`apiService.caBundle` can also be the server certificate.

To use this method, please setup your values file like this:

```yaml
apiService:
insecureSkipTLSVerify: false
caBundle:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls:
type: existingSecret
existingSecret:
name: metrics-server-existing
```
8 changes: 8 additions & 0 deletions charts/metrics-server/ci/tls-certManager-values.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
args:
- --kubelet-insecure-tls

apiService:
insecureSkipTLSVerify: false

tls:
type: cert-manager
8 changes: 8 additions & 0 deletions charts/metrics-server/ci/tls-helm-values.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
args:
- --kubelet-insecure-tls

apiService:
insecureSkipTLSVerify: false

tls:
type: helm
47 changes: 43 additions & 4 deletions charts/metrics-server/templates/apiservice.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,56 @@
{{ $commonName := include "metrics-server.fullname" . }}
{{ $ns := .Release.Namespace }}
{{- $altNames := list -}}
{{- $altNames = append $altNames (printf "%s.%s" $commonName $ns) -}}
{{- $altNames = append $altNames (printf "%s.%s.svc" $commonName $ns) -}}
{{- $altNames = append $altNames (printf "%s.%s.svc.%s" $commonName $ns .Values.tls.certManager.clusterDomain) -}}
{{- $certs := genSelfSignedCert $commonName nil $altNames 36500 -}}
{{- if or (eq .Values.tls.type "secret") (eq .Values.tls.type "helm") }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "metrics-server.fullname" . }}
{{- with .Values.tls.secret.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "metrics-server.labels" . | nindent 4 }}
{{- with .Values.tls.secret.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
type: Opaque
data:
{{- if eq .Values.tls.type "secret" }}
tls.crt: {{ .Values.tls.secret.crt | b64enc | quote }}
tls.key: {{ .Values.tls.secret.key | b64enc | quote }}
{{- else }}
tls.crt: {{ $certs.Cert| b64enc | quote }}
tls.key: {{ $certs.Key | b64enc | quote }}
{{- end }}
{{- end }}
---
{{- if .Values.apiService.create -}}
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
labels:
{{- include "metrics-server.labels" . | nindent 4 }}
{{- with .Values.apiService.annotations }}
{{- if or .Values.apiService.annotations .Values.tls.certManager.addInjectorAnnotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- if and (eq .Values.tls.type "cert-manager") .Values.tls.certManager.addInjectorAnnotations }}
cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ include "metrics-server.fullname" . }}
{{- end }}
{{- with .Values.apiService.annotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
spec:
{{- with .Values.apiService.caBundle }}
caBundle: {{ b64enc . }}
{{- if eq .Values.tls.type "helm" }}
caBundle: {{ $certs.Cert | b64enc }}
{{- else if and .Values.apiService.caBundle (ne .Values.tls.type "cert-manager") }}
caBundle: {{ .Values.apiService.caBundle | b64enc }}
{{- end }}
group: metrics.k8s.io
groupPriorityMinimum: 100
Expand Down
47 changes: 47 additions & 0 deletions charts/metrics-server/templates/certificate.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{{- if eq .Values.tls.type "cert-manager" }}
{{- if not .Values.tls.certManager.existingIssuer.enabled }}
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
annotations:
{{- toYaml .Values.additionalAnnotations | nindent 4 }}
name: {{ include "metrics-server.fullname" . }}-issuer
namespace: {{ .Release.Namespace }}
spec:
selfSigned: {}
{{- end }}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ include "metrics-server.fullname" . }}
namespace: {{ .Release.Namespace }}
spec:
commonName: {{ include "metrics-server.fullname" . }}
dnsNames:
- {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}
- {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}.svc
- {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.tls.certManager.clusterDomain }}
secretName: {{ include "metrics-server.fullname" . }}
usages:
- server auth
- client auth
privateKey:
algorithm: RSA
size: 2048
{{- with .Values.tls.certManager.duration }}
duration: {{ . }}
{{- end }}
{{- with .Values.tls.certManager.renewBefore }}
renewBefore: {{ . }}
{{- end }}
issuerRef:
{{- if .Values.tls.certManager.existingIssuer.enabled }}
name: {{ .Values.tls.certManager.existingIssuer.name }}
kind: {{ .Values.tls.certManager.existingIssuer.kind }}
{{- else }}
name: {{ include "metrics-server.fullname" . }}-issuer
kind: Issuer
{{- end }}
group: cert-manager.io
{{- end }}
18 changes: 18 additions & 0 deletions charts/metrics-server/templates/deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ spec:
{{- if .Values.metrics.enabled }}
- --authorization-always-allow-paths=/metrics
{{- end }}
{{- if ne .Values.tls.type "metrics-server" }}
- --tls-cert-file=/tmp/tls-certs/tls.crt
- --tls-private-key-file=/tmp/tls-certs/tls.key
{{- end }}
{{- range .Values.args }}
- {{ . }}
{{- end }}
Expand All @@ -80,6 +84,11 @@ spec:
volumeMounts:
- name: tmp
mountPath: /tmp
{{- if ne .Values.tls.type "metrics-server" }}
- mountPath: /tmp/tls-certs
name: certs
readOnly: true
{{- end }}
{{- with .Values.extraVolumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
Expand Down Expand Up @@ -126,6 +135,15 @@ spec:
configMap:
name: {{ include "metrics-server.addonResizer.configMap" . }}
{{- end }}
{{- if ne .Values.tls.type "metrics-server" }}
- name: certs
secret:
{{- if and (eq .Values.tls.type "existingSecret") .Values.tls.existingSecret.name }}
secretName: {{ .Values.tls.existingSecret.name }}
{{- else }}
secretName: {{ include "metrics-server.fullname" . }}
{{- end }}
{{- end }}
{{- with .Values.extraVolumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
Expand Down
4 changes: 4 additions & 0 deletions charts/metrics-server/templates/extra-manifests.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{{ range .Values.extraObjects }}
---
{{ tpl (toYaml .) $ }}
{{ end }}
81 changes: 81 additions & 0 deletions charts/metrics-server/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,84 @@ topologySpreadConstraints: []
deploymentAnnotations: {}

schedulerName: ""

tls:
# Either use `metrics-server`, `secret`, `existingSecret`, `cert-manager` or `helm`. Details:
# `metrics-server` : Metrics-server will generate self-signed certs
# `secret` : Create a new secret with user-provided key and certificate
# `existingSecret` : Reuse an existing secret. No new secret will be created
# `cert-manager` : Use cert-manager.io to create and maintain the certificate for you
# `helm` : Helm will generate the certificate
type: "metrics-server"

certManager:
# Kubernetes cluster domain. Used to configure Subject Alt Names for the certificate
clusterDomain: cluster.local
# Automatically add the cert-manager.io/inject-ca-from annotation to the APIService resource.
# See https://cert-manager.io/docs/concepts/ca-injector
addInjectorAnnotations: true
existingIssuer:
# Use an existing cert-manager issuer
enabled: false
# Kind of the existing cert-manager issuer
kind: "Issuer"
# Name of the existing cert-manager issuer
name: "my-issuer"
# Set the requested duration (i.e. lifetime) of the Certificate.
# See https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec
duration: ""
# How long before the currently issued certificate’s expiry cert-manager should renew the certificate.
# See https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec
renewBefore: ""
# Add extra annotations to the Certificate resource
annotations: {}
# Add extra labels to the Certificate resource
labels: {}

existingSecret:
# Name of the existing Secret to use for TLS
name: ""

secret:
# Add extra annotations to the Secret resource
annotations: {}
# Add extra labels to the Secret resource
labels: {}
# The server certificate to use for metrics-server. Use PEM format
crt: ""
# The private key of the certificate to use for metrics-server. Use PEM format.
key: ""

# List of extra manifests to deploy. Will be passed through `tpl` to support templating
extraObjects: []
# - apiVersion: external-secrets.io/v1beta1
# kind: ExternalSecret
# metadata:
# name: my-external-secret
# namespace: '{{ .Release.Namespace }}'
# spec:
# secretStoreRef:
# kind: ClusterSecretStore
# name: my-secret-store
# target:
# name: my-kubernetes-secret
# data:
# - secretKey: secretKey
# remoteRef:
# key: /path/to/my-secret

# - apiVersion: secrets.hashicorp.com/v1beta1
# kind: VaultStaticSecret
# metadata:
# name: my-external-vault-secret
# namespace: '{{ .Release.Namespace }}'
# spec:
# type: kv-v2
# mount: kvv2
# path: webapp/config
# destination:
# name: secretkv
# create: true
# refreshAfter: 30s
# refreshAfter: 30s
# vaultAuthRef: static-auth

0 comments on commit 0254b50

Please sign in to comment.