Replies: 7 comments 8 replies
-
|
I think that this is a good approach, the "State of Kubernetes Security Report" shows that most incidents are caused by misconfiguration (page 4). By adding this functionality, cdk8s can help users avoid those pitfalls. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, my gut feeling matches the UNIX philosophy here: don't overcomplicated this tool, let it do one thing well. Chaining validators behind it (as part of CI pipelines, before deployment!) should be easy -- do those accept a set of YAML files on STDIN, for instance? |
Beta Was this translation helpful? Give feedback.
-
|
@reitzig we were debating this quite a bit internally. Eventually we considered that, while chaining validators is technically easy, it can be procedurally complicated in large organizations. And streamlining this process within cdk8s can really help. Have a look at our RFC for a comprehensive rundown of our thought process. THanks! |
Beta Was this translation helpful? Give feedback.
-
|
validation tools to prevent pitfalls is great, but because with every new K8s release, there are new properties that are deprecated, I would also consider incorporating a tool to validate the K8s schema: |
Beta Was this translation helpful? Give feedback.
-
|
That's not necessarily enough. |
Beta Was this translation helpful? Give feedback.
-
|
@rrey-aviatrix Can you provide the code snippet you used? I believe what you're describing is a situation where the schema defines two optional properties, because either one can be used, but not both. For example, the schema for a volume is: "io.k8s.api.core.v1.Volume": {
"description": "Volume represents a named volume in a pod that may be accessed by any container in the pod.",
"properties": {
"awsElasticBlockStore": {
"$ref": "#/definitions/io.k8s.api.core.v1.AWSElasticBlockStoreVolumeSource",
"description": "AWSElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore"
},
"azureDisk": {
"$ref": "#/definitions/io.k8s.api.core.v1.AzureDiskVolumeSource",
"description": "AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod."
},
...
...
}
}You can either specify The "not both" part is not encoded in the schema so there's no way for cdk8s to know that. These types of problems is where cdk8s-plus really helps because it provides an API that only allows you to specify one of them. @eyarz Are the schema validation tools you know of able to catch these types of errors as well? or do they only validate the YAML is a legal k8s manifest? |
Beta Was this translation helpful? Give feedback.
-
|
@iliapolo yes it is the same kind of issue with the Ingress backend. Here is a snippet The synth is quiet and when deploying: So having a post synthesis validation that could raise this kind of issues in the CI before any attempt to deploy would be nice. |
Beta Was this translation helpful? Give feedback.
-
|
An interesting fact about K8s schema - not all validations are defined in the schema. There are some validations that you will expect to be n the schema but are actually validated by a Go script before the manifest is deployed (on the server side). For example, validating labels value is not part of the schema definition. Therefore, existing schema validation tools (like Kubeconform & Kubeval) will not catch this type of misconfigurations, but Datree has a set of built-in rules (out-of-the-box) to catch them (e.g. Ensure workload has valid label values) @iliapolo if the cdk8s API is generated from the schema definitions, it can't create conflicts if I'm using (in my cluster) a different K8s version? |
Beta Was this translation helpful? Give feedback.
-
|
@eyarz You mean conflicts between two different versions? The version cdk8s uses is fully controlled by the user - so if your cluster is 1.22.0, you should tell cdk8s to generate the API for version 1.22.0. This is an option in the cdk8s configuration file. |
Beta Was this translation helpful? Give feedback.
-
|
After some more time to think about this, summarizing what I came up with so far: WhyI think the need for Kubernetes manifest validation in general has already been substantiated, so in this section I want to answer the question: Why should cdk8s in particular offer manifest validation mechanisms? One could argue that, since cdk8s produces native k8s manifests, users can simply invoke any validation tool they want after running cdk8s synth, and that there really is no need for explicit cdk8s support for it. While this is technically true, I think cdk8s is well positioned to make setting up this workflow much easier. Before we dive into why, I'd like to lay out an assumption we should agree on: “In large organizations, validation policies are defined by a central security & compliance team, while manifests are written by all development teams“ Note that we are not focused on deploy time here, i.e preventing misconfiguration from landing in your cluster. So, without explicit cdk8s support, how can organizations ensure their dev teams are writing compliant manifests? 1) Local editingThere are basically two ways of achieving this:
Guides are great as a source of information, but they are not a mechanism. They will most likely Wrapper tools are also good, but like guides, there's really no way of ensuring developers use them. They also pose an additional maintenance burden on a central team, that might not have the necessary skills to create such tools. Also, if you think about, cdk8s is already a wrapper tool for authoring manifests, and developers are already using it, so doesn't it make sense for cdk8s to offer this capability? Essentially, without cdk8s, the authoring of manifests is inherently decoupled from its validation, and thus will 2) Pull RequestAn organization can add a validation command to the CI pipelines that validate pull requests. This is not so easy to achieve, as CI pipelines can greatly vary between teams. Composing → Publishing → Reusing workflows can be fairly complicated. Also, it delays the validation by the latency of the pipeline, and introduces another failure point for the feedback loop. To summarize, given developers are already using cdk8s to author manifests locally, it is the perfect
|
Beta Was this translation helpful? Give feedback.
-
WhatThe proposal here is to allow cdk8s to integrate with third-party k8s manifest validation tools. For example, given the following cdk8s code: export class MyChart extends Chart {
constructor(scope: Construct, id: string) {
super(scope, id);
new k8s.KubeDeployment(this, 'Deployment', {
spec: {
template: {
containers: [{ image: 'nginx', name: 'main' }]
}
}
});
}
}
const app = new App();
new MyChart(app, 'Chart');
app.synth();Along with configuring* datree as the validation tool, running cdk8s synth should produce something similar to: ❌ Ensure each container has a configured CPU request [1 occurrence]
- construct: Chart/Deployment (kind: Deployment)
💡 Missing property object `requests.cpu` - value should be within the accepted boundaries recommended by the organization
❌ Ensure each container image has a pinned (tag) version [1 occurrence]
- construct: Chart/Deployment (kind: Deployment)
💡 Incorrect value for key `image` - specify an image version to avoid unpleasant "version surprises" in the future
RequirementsOpen SystemThere are many validation tools out there. We would like cdk8s to be able to support all of them, without requiring code changes in the core library for every new tool. Ideally, these integrations will be maintained by the authors of the validation tools, and not by the cdk8s team. Construct AwareBy default, the validation tools operate on k8s resources. The reports they generate usually point to specific resources by name. However, when writing cdk8s applications, providing a resource name is actually discouraged, so the report needs to point to a specific construct instead. In addition, because of the imperative nature of cdk8s, it’s not always easy to figure out where exactly in the code was a construct even created. We would like the report to also provide a sort of traceback to help developers locate the specific code being used. Distributable ConfigAs mentioned earlier, we are operating under the assumption that validation config (e.g which tool, what policy) is defined by central compliance and security teams, not by the developers themselves. This means that these teams need a way to publish common configuration that all developer teams can then easily consume. |
Beta Was this translation helpful? Give feedback.
-
HowDistributable ConfigValidations will be declared in the cdk8s.yaml configuration file, and the cdk8s CLI will try to load and execute them. For example, a configuration will look something like: app: node main.js
language: typescript
imports:
- k8s
validations:
- http://mycompany.com/cdk8s-validations.yamlWhere - package: datree-cdk8s-plugin
class: DatreeValidation
config:
policyUrl: http://mycompany.com/k8s-datree-policy.yaml
- package: checkov-cdk8s-plugin
class: CheckovValidation
config:
rulesUrl: http://mycompany.com/k8s-checkov-policy.yaml
Every manifest generated as part of a cdk8s app that uses this config, will undergo both datree and checkov validations, based on the configuration properties provided by the central team. This way, central teams only maintain the validations config and policies, and developers only need to point their config file to the appropriate validations URL. At this point you might be wondering - how would organizations ensure developers actually configure the validations property to point to the right location? What prevents them from simply omitting it? Well, nothing really, when developers are writing manifests, there’s always going to be some setup they need to do. Our goal is to minimize it, and make it as easy as possible. The way I see this playing out is:
FAQ Q. Why not inline the validations config in the cdk8s.yaml file? Why this level of indirection? Open SystemFor the validation mechanism to be an open system, we need to create a sort of plugin mechanism. As already alluded in the previous section, third-parties should be able to implement and publish a validation plugin, and cdk8s should be able to load and execute it. Essentially what this means is that we need to define an interface, that third-parties can implement: export interface Validation {
/**
* Run the validation tool and produce a report.
*/
validate(manifests: string[]): Violation[];
}Where the /**
* Represents a distinct violation in the manifest.
*/
export interface Violation {
/**
* The name of resource exhibiting the violation.
*/
readonly resourceName: string;
/**
* The violation message.
*/
readonly message: string;
/**
* The manifest file this violation originated in.
*/
readonly manifestPath: string;
}With these interfaces in-place, this is how a datree implementation would potentially look like: import { Validation, Violation } from 'cdk8s-cli';
export interface DatreeValidationProps {
/**
* URL for the policy to use.
*/
readonly policyUrl: string;
}
export class DatreeValidation implements Validation {
private readonly policyUrl: string;
constructor(props: DatreeValidationProps) {
this.policyUrl = props.policyUrl;
}
public validate(manifests: string[]): Violation[] {
const report: Violation[] = [];
for (const manifest of manifests) {
const out = execSync(`datree test --output json ${manifest}`);
report.push(...parseOutput(out));
}
return report;
}
} During cdk8s synth, the CLI will look at the validations config property, download it from the remote source if necessary, and proceed to load and execute the plugins. FAQ Q. Who is responsible for datree CLI to exist on the system? the user? cdk8s? Q. Should cdk8s synth fail if the validation results in violations? Construct AwareThe ability to traceback to constructs world, will be enabled by adding a few annotations to the manifests generated by cdk8s. More concretely, we add cdk8s.io/construct.path and cdk8s.io/construct.traceback annotations: apiVersion: apps/v1
kind: Deployment
metadata:
name: chart-deployment-c8a3b439
annotations:
cdk8s.io/construct.path: Chart/Deployment
cdk8s.io/construct.traceback:
- deployment.ts:25
- main.ts:15
...
...This should be fairly easily achievable by adding this logic in our base ApiObject class. Once we have it, we can find the violating resource by name, since the name is provided in the report, and then extract both the traceback and path from these annotations, to produce the final report we show to users. |
Beta Was this translation helpful? Give feedback.
-
Future PossibilitiesAdvanced in-memory validationsThe mechanism described here suggests the validations are executed by the CLI. We could however, run the validations in the framework, as part of App.synth. This is somewhat more involved, but can allow writing more advanced plugins and validations, since they can have access to the entire construct tree. Organizational DefaultsSimilarly to validations, organizations also want to define their own org specific defaults that should be used in manifests. |
Beta Was this translation helpful? Give feedback.
-
|
Following are the list of tasks to implement this capability: |
Beta Was this translation helpful? Give feedback.
-
Ability to define manifest validators to be executed after synthesis
There are plenty of pitfalls users may fall into when defining their manifests. Validation tools exist that will help users avoid those. For example:
Since cdk8s allows us to run code during manifest generation, we can integrate these tools into the process, without users having to invoke them explicitly. Our init templates can offer these validations by default, and allow users to opt out, or vise versa.
Add a key in
cdk8s.yamlto describe what validation should take place:Open questions
Beta Was this translation helpful? Give feedback.
All reactions