Merge pull request #182 from cloudbees/cbci-agents-sec-docker

[Core & Blueprints, 02-at-scale]: Adding Container Registry compatibility
cloudbees · Aug 2, 2024 · 5fcd280 · 5fcd280
2 parents 531531f + 4b8ec00
commit 5fcd280
Show file tree

Hide file tree

Showing 43 changed files with 1,041 additions and 729 deletions.
diff --git a/.cloudbees/workflows/my-test.yaml b/.cloudbees/workflows/my-test.yaml
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -23,7 +23,7 @@ repos:
     - id: check-vcs-permalinks
 
 - repo: https://github.com/antonbabenko/pre-commit-terraform
-  rev: v1.81.0
+  rev: v1.92.0
   hooks:
     - id: terraform_fmt
       name: Format Terraform Configuration
@@ -33,7 +33,7 @@ repos:
         - --hook-config=--retry-once-with-cleanup=true
     - id: terraform_tflint
       args:
-        - '--args=--only=terraform_deprecated_interpolation'
+        #- '--args=--only=terraform_deprecated_interpolation'
         - '--args=--only=terraform_deprecated_index'
         - '--args=--only=terraform_unused_declarations'
         - '--args=--only=terraform_comment_syntax'

diff --git a/README.md b/README.md
@@ -19,8 +19,8 @@
 
 The CloudBees CI [AWS partner add-on](https://aws-ia.github.io/terraform-aws-eks-blueprints-addons/main/aws-partner-addons/) streamlines the adoption and experimentation of CloudBees CI enterprise features by:
 
-- Encapsulating the deployment of [CloudBees CI on modern platforms in AWS EKS](https://docs.cloudbees.com/docs/cloudbees-ci/latest/eks-install-guide/installing-eks-using-helm#_configuring_your_environment) into a Terraform module.
-- Providing a series of [blueprints](https://github.com/cloudbees/terraform-aws-cloudbees-ci-eks-addon/tree/main/blueprints) that implement the CloudBees CI add-on module for use with [Amazon EKS blueprints for Terraform](https://aws-ia.github.io/terraform-aws-eks-blueprints-addons/main/), which are aligned with the [EKS Best Practices Guides](https://aws.github.io/aws-eks-best-practices/).
+- Encapsulating the deployment of [CloudBees CI on modern platforms in AWS EKS](https://docs.cloudbees.com/docs/cloudbees-ci/latest/eks-install-guide/installing-eks-using-helm#_configuring_your_environment) and additional k8s resources into a Terraform module.
+- Providing a series of opinionated [blueprints](https://github.com/cloudbees/terraform-aws-cloudbees-ci-eks-addon/tree/main/blueprints) that implement the CloudBees CI add-on module for use with [Amazon EKS blueprints for Terraform](https://aws-ia.github.io/terraform-aws-eks-blueprints-addons/main/), which are aligned with the [EKS Best Practices Guides](https://aws.github.io/aws-eks-best-practices/).
 
 ## Usage
 
@@ -95,10 +95,13 @@ The CloudBees CI add-on uses `helms release` for its resources definition, makin
 | cert_arn | AWS Certificate Manager (ACM) certificate for Amazon Resource Names (ARN). | `string` | n/a | yes |
 | hosted_zone | Amazon Route 53 hosted zone name. | `string` | n/a | yes |
 | trial_license | CloudBees CI trial license details for evaluation. | `map(string)` | n/a | yes |
-| create_k8s_secrets | Create the Kubernetes secret cbci-secrets and mount into the Operation Center /run/secrets/ path to be consumed by CloudBees CasC. | `bool` | `false` | no |
+| casc_secrets_file | Secrets .yml file path containing the names:values secrets. It is required when create_casc_secrets is enabled. | `string` | `"secrets-values.yml"` | no |
+| create_casc_secrets | Create a Kubernetes basic secret for CloudBees Configuration as Code (cbci-sec-casc) and mount it into the Operation Center /var/run/secrets/cbci. | `bool` | `false` | no |
+| create_reg_secret | Create a Kubernetes dockerconfigjson secret for Container Registry authentication (cbci-sec-reg) for CI builds agents. | `bool` | `false` | no |
 | helm_config | CloudBees CI Helm chart configuration. | `any` | <pre>{<br>  "values": [<br>    ""<br>  ]<br>}</pre> | no |
-| k8s_secrets | Secrets .yml file as a string containing the names:values secrets. It is required when create_k8s_secrets is enabled. | `string` | `"secrets-values.yml"` | no |
 | prometheus_target | Creates a service monitor to discover the CloudBees CI Prometheus target dynamically. It is designed to be enabled with the AWS EKS Terraform Addon Kube Prometheus Stack. | `bool` | `false` | no |
+| reg_secret_auth | Registry server authentication details for cbci-sec-reg secret. It is required when create_reg_secret is enabled. | `map(string)` | <pre>{<br>  "email": "[email protected]",<br>  "password": "changeme1234",<br>  "server": "my-registry.acme:5000",<br>  "username": "foo"<br>}</pre> | no |
+| reg_secret_ns | Agent Namespace to allocate cbci-sec-reg secret. It is required when create_reg_secret is enabled. | `string` | `"cbci"` | no |
 
 ### Outputs
 
@@ -111,7 +114,8 @@ The CloudBees CI add-on uses `helms release` for its resources definition, makin
 | cbci_oc_ing | Operations center Ingress for the CloudBees CI add-on. |
 | cbci_oc_pod | Operations center pod for the CloudBees CI add-on. |
 | cbci_oc_url | Operations center URL for the CloudBees CI add-on using a subdomain and certificates. |
-| cbci_secrets | Optional. Kubernetes secrets name for CloudBees CI. |
+| cbci_sec_casc | Optional. Kubernetes secrets name for CloudBees CI Casc. |
+| cbci_sec_registry | Optional. Kubernetes secrets name for CloudBees CI agents to autheticate to registry. |
 | merged_helm_config | (merged) Helm configuration for CloudBees CI. |
 <!-- END_TF_DOCS -->
 

diff --git a/blueprints/01-getting-started/README.md b/blueprints/01-getting-started/README.md
@@ -19,7 +19,7 @@ This blueprint presents the minimum setup to run CloudBees CI on Amazon EKS; one
   - Instance type: [Graviton Processor](https://aws.amazon.com/ec2/graviton/) family.
   - Amazon Machine Image (AMI) type: [Amazon EKS-Optimized Amazon Linux 2023](https://aws.amazon.com/blogs/containers/amazon-eks-optimized-amazon-linux-2023-amis-now-available/)
 
-### Kubernetes cluster
+### Workloads
 
 ![K8sApps](img/getting-started.k8s.drawio.svg)
 

diff --git a/blueprints/02-at-scale/.auto.tfvars.example b/blueprints/02-at-scale/.auto.tfvars.example
@@ -7,8 +7,11 @@ trial_license = { # Required. CloudBees CI Trial license details for evaluation.
   company = "Acme Inc."
 }
 
-gh_user = "exampleUser" # Required. Default values can be used for demo.
-gh_token = "ExampleToken1234" # Required. Default values can be used for demo.
+dh_reg_secret_auth = { # Required.
+  username = "foo"
+  password = "d0ckerPass12"
+  email    = "[email protected]"
+}
 
 # tags = {  # Optional. Tags for the resources created. Default set to empty. Shared among all.
 #   "cb-owner" : "team-services"

diff --git a/blueprints/02-at-scale/README.md b/blueprints/02-at-scale/README.md
@@ -53,7 +53,7 @@ This blueprint divides scalable node groups for different types of workloads:
 
 ![Architecture](img/at-scale.architect.drawio.svg)
 
-### Kubernetes cluster
+### Workloads
 
 ![K8sApps](img/at-scale.k8s.drawio.svg)
 
@@ -68,8 +68,7 @@ This blueprint divides scalable node groups for different types of workloads:
 | trial_license | CloudBees CI trial license details for evaluation. | `map(string)` | n/a | yes |
 | aws_region | AWS region to deploy resources to. It requires at minimun 3 AZs. | `string` | `"us-west-2"` | no |
 | ci | Running in a CI service versus running locally. False when running locally, true when running in a CI service. | `bool` | `false` | no |
-| gh_token | GitHub token for the CloudBees operations center credential GH-User-token, that is created via CloudBees CasC. | `string` | `"ExampleToken1234"` | no |
-| gh_user | GitHub user for the CloudBees operations center credential GH-User-token, that is created via CloudBees CasC. | `string` | `"exampleUser"` | no |
+| dh_reg_secret_auth | Docker Hub Registry server authentication details for cbci-sec-reg secret. | `map(string)` | <pre>{<br>  "email": "[email protected]",<br>  "password": "changeme1234",<br>  "username": "foo"<br>}</pre> | no |
 | suffix | Unique suffix to assign to all resources. When adding the suffix, changes are required in CloudBees CI for the validation phase. | `string` | `""` | no |
 | tags | Tags to apply to resources. | `map(string)` | `{}` | no |
 
@@ -81,9 +80,10 @@ This blueprint divides scalable node groups for different types of workloads:
 | aws_backup_efs_protected_resource | AWS description for the Amazon EFS drive that is used to back up protected resources. |
 | aws_logstreams_fluentbit | AWS CloudWatch log streams from Fluent Bit. |
 | cbci_agent_linuxtempl_events | Retrieves a list of events related to Linux template agents. |
+| cbci_agent_sec_reg | Retrieves the container registry secret deployed in the agents namespace. |
 | cbci_agent_windowstempl_events | Retrieves a list of events related to Windows template agents. |
 | cbci_agents_pods | Retrieves a list of agent pods running in the agents namespace. |
-| cbci_controller_b_ws_cache_build | team-b hibernation monitor endpoint to the build workspace cache. It expects CBCI_ADMIN_TOKEN as the environment variable. |
+| cbci_controller_b_s3_build | team-b hibernation monitor endpoint to the build s3-WScacheAndArtifacts. It expects CBCI_ADMIN_TOKEN as the environment variable. |
 | cbci_controller_c_hpa | team-c horizontal pod autoscaling. |
 | cbci_controller_c_windows_node_build | team-c hibernation monitor endpoint to the Windows build nodes. It expects CBCI_ADMIN_TOKEN as the environment variable. |
 | cbci_controllers_pods | Operations center pod for the CloudBees CI add-on. |
@@ -117,6 +117,13 @@ This blueprint divides scalable node groups for different types of workloads:
 | vpc_arn | VPC ID. |
 <!-- END_TF_DOCS -->
 
+## Prerequisites
+
+This blueprint uses [DockerHub](https://hub.docker.com/) as a Container Registry Service. Then, an existing DockerHub account is required (username, password and email).
+
+> [!TIP]
+> Use `docker login` to validate username and password.
+
 ## Deploy
 
 When preparing to deploy, you must complete the following steps:
@@ -148,14 +155,13 @@ Once the resources have been created, a `kubeconfig` file is created in the [/k8
 
 1. Complete the steps to [validate CloudBees CI](../01-getting-started/README.md#cloudbees-ci), if you have not done so already.
 
-2. Authentication in this blueprint is based on LDAP using the `cn` user (available in [k8s/openldap-stack-values.yml](./k8s/openldap-stack-values.yml)) and the global password. The authorization level defines a set of permissions configured using [RBAC](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-secure-guide/rbac). Additionally, the operations center and controller use [single sign-on (SS0)](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-secure-guide/using-sso). Issue the following command to retrieve the global password:
+2. Authentication in this blueprint is based on LDAP using the `cn` user (available in [k8s/openldap-stack-values.yml](./k8s/openldap-stack-values.yml)) and the global password. The authorization level defines a set of permissions configured using [RBAC](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-secure-guide/rbac). Additionally, the operations center and controller use [single sign-on (SS0)](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-secure-guide/using-sso), including a [fallback mechanism](https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/operations-center/how-ldap-plugin-works-on-cjoc-sso-context) that is enabled by default. Issue the following command to retrieve the global password (valid for all users):
 
    ```sh
    eval $(terraform output --raw global_password)
    ```
 
-   > [!NOTE]
-   > There are differences in CloudBees CI permissions and folder restrictions when signed in as a user of the Admin group versus the Development group. For example, only Admin users have access to the agent validation jobs.
+   There are differences in CloudBees CI permissions and folder restrictions when signed in as a user of the Admin group versus the Development group. For example, only Admin users have access to the agent validation jobs.
 
 3. CasC is enabled for the [operations center](https://docs.cloudbees.com/docs/cloudbees-ci/latest/casc-oc/) (`cjoc`) and [controllers](https://docs.cloudbees.com/docs/cloudbees-ci/latest/casc-controller/) (`team-b` and `team-c-ha`). `team-a` is not using CasC, to illustrate the difference between the two approaches. Issue the following command to verify that all controllers are Running:
 
@@ -175,10 +181,26 @@ Once the resources have been created, a `kubeconfig` file is created in the [/k8
 
 ##### Kubernetes Secret
 
-The secrets key/value file defined in [k8s/secrets-values.yml](k8s/secrets-values.yml) is converted into a Kubernetes secret (`cbci-secrets`) and mounted into /run/secrets/ for Operation Center and Controllers to be consumed via CloudBees Casc. See [Configuration as Code - Handling Secrets - Kubernetes Secrets](https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/docs/features/secrets.adoc#kubernetes-secrets) for more information. Beyond the CloudBees CI Addon, Kubernetes secrets can be also created via [External Secret Operators](https://aws-ia.github.io/terraform-aws-eks-blueprints-addons/main/addons/external-secrets/).
+This blueprint uses a couple of Kubernetes secrets for different purposes.
 
 > [!NOTE]
-> Kubernetes secrets can be also be retrived as Jenkins Credentials via using the [Kubernetes Credentials Provider plugin](https://jenkinsci.github.io/kubernetes-credentials-provider-plugin/).
+> - Beyond the CloudBees CI Addon (for demo purposes), Kubernetes secrets can be managed via [External Secret Operators](https://aws-ia.github.io/terraform-aws-eks-blueprints-addons/main/addons/external-secrets/).
+> - Kubernetes secrets could be also be retrived as Jenkins Credentials via using the [Kubernetes Credentials Provider plugin](https://jenkinsci.github.io/kubernetes-credentials-provider-plugin/).
+
+###### Casc Secrets
+
+The secrets key/value file defined in [k8s/secrets-values.yml](k8s/secrets-values.yml) is converted into a Kubernetes secret (`cbci-sec-casc`) and mounted into /run/secrets/ for Operation Center and Controllers to be consumed via CloudBees Casc. See [Configuration as Code - Handling Secrets - Kubernetes Secrets](https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/docs/features/secrets.adoc#kubernetes-secrets) for more information.
+
+###### Container Registry Secrets
+
+DockerHub authentication is stored as Kubernetes secrets (`cbci-agent-sec-reg`) and mounted to [Kaniko agent containers](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/using-kaniko) to build and push images to this registry. The secret is created using the `dh_reg_secret_auth` variable.
+
+```sh
+   eval $(terraform output --raw cbci_agent_sec_reg)
+```
+
+> [!NOTE]
+> ECR authentication is done via instance profile connected to `build-linux-spot` Node pools.
 
 ##### HashiCorp Vault
 
@@ -203,6 +225,8 @@ HashiCorp Vault is used as a credential provider for CloudBees CI Pipelines in t
 
 #### Builds
 
+##### Build Node Pools
+
 1. For the following validations, builds will be triggered remotely. Start by issuing the following command to retrieve an [API token](https://docs.cloudbees.com/docs/cloudbees-ci-api/latest/api-authentication) for the `admin_cbci_a` user with the correct permissions for the required actions:
 
    ```sh
@@ -219,25 +243,27 @@ HashiCorp Vault is used as a credential provider for CloudBees CI Pipelines in t
 
 2. Once you have retrieved the API token, issue the following commands to trigger builds using the [POST queue for hibernation API endpoint](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/managing-controllers#_post_queue_for_hibernation). If successful, an `HTTP/2 201` response is returned, indicating the REST API call has been correctly received by the CloudBees CI controller.
 
-- For Linux node pools use:
+   - For Linux node pools use:
 
-   ```sh
-   eval $(terraform output --raw cbci_controller_b_ws_cache_build)
-   ```
+      ```sh
+      eval $(terraform output --raw cbci_controller_b_s3_build)
+      ```
 
-   It triggers `ws-cache` pipeline from `team-b` controller. This pipeline uses [CloudBees Workspace Caching](https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/cloudbees-cache-step). Once the second build is complete, you can find the read cache operation at the beginning of the build logs and the write cache operation at the end of the build logs.
+      It triggers `s3-WScacheAndArtifacts` pipeline from `team-b` controller. This pipeline validates a couple of s3 integrations in parallel: one for [CloudBees Workspace Caching](https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/cloudbees-cache-step) (using `linux-mavenAndKaniko-L`) and another for [s3 Artifact Manager](https://plugins.jenkins.io/artifact-manager-s3/) (using `linux-mavenAndKaniko-XL`).
 
-   Note that this pipeline uses the On-Demand Linux Node Pool but there is also Spot Linux Node Pool available.
+      Once the second build is complete, you can find the read cache operation at the beginning of the build logs and the write cache operation at the end of the build logs.
 
-- For Windows node pool use:
+      The `linux-mavenAndKaniko-L` agent template is deployed over On-demand Linux Nodes that have smaller instance types vs the `linux-mavenAndKaniko-XL` template that is deployed over Spot Linux Nodes that have defined larger instance types.
 
-   ```sh
-   eval $(terraform output --raw cbci_controller_c_windows_node_build)
-   ```
+   - For Windows node pool use:
 
-   It triggers `windows-build-nodes` pipeline from `team-c-ha` controller.
+      ```sh
+      eval $(terraform output --raw cbci_controller_c_windows_node_build)
+      ```
 
-   Note that the first build for a new Windows image container can take up to 10 minutes to run; subsequent builds should take seconds to run. This behavior can be improved as explained in the section [Architecture](#architecture).
+      It triggers `windows-build-nodes` pipeline from `team-c-ha` controller.
+
+      Note that the first build for a new Windows image container can take up to 10 minutes to run; subsequent builds should take seconds to run. This behavior can be improved as explained in the section [Architecture](#architecture).
 
 3. Right after triggering the builds, issue the following to validate pod agent provisioning to build the pipeline code:
 
@@ -247,6 +273,21 @@ HashiCorp Vault is used as a credential provider for CloudBees CI Pipelines in t
 
 4. Check build logs by signing in to the `team-b` and `team-c-ha` controllers, respectively. Navigate to the pipeline jobs and select the first build, indicated by the `#1` build number. [CloudBees Pipeline Explorer](https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/cloudbees-pipeline-explorer-plugin) is enabled as a default.
 
+##### Container Registry
+
+This blueprints uses a couple of container registries for different purposes.
+
+- The Public Registry uses DockerHub.
+- The Private Registry uses AWS ECR.
+
+> [!NOTE]
+> Other Container Registry services can be used for the same purporses.
+
+Access with admin role to CloudBees CI to `team-b` or `team-c-ha` to run the pipeline _admin_ > _validations_ > _kaniko_ enter as parameters an existing DockerHub Organization and an existing AWS ECR Repository to test that Build and Push towards every repository works fine.
+
+> [!NOTE]
+> Besides Kaniko, there are [other alternitives tools](https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/using-kaniko#_alternatives).
+
 #### Back up and restore
 
 For backup and restore operations, you can use the [preconfigured CloudBees CI Cluster Operations job](#create-daily-backups-using-a-cloudbees-ci-cluster-operations-job) to automatically perform a daily backup, which can be used for Amazon EFS and Amazon EBS storage.