FR: CSR (Cloud Source Repositories) EOL June 2024 - replace the default path for version control system and CICD tool

[fmichaelobrien]

TL;DR

CSR is undergoing deprecation in favour of SSM

• https://cloud.google.com/source-repositories/docs/authentication
• https://cloud.google.com/secure-source-manager/docs/overview

Impact to CICD

• will affect the CB/CSR default option in the TEF LZ - primarily 0-bootstrap starting with the following gcloud clone
• https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/scripts/push-to-repo.sh#L32
• CB/Cloud Build will continue to be used for the pipeline
• will not affect incoming ADO option in FR: Add Azure DevOps CI/CD repository and pipeline support option #1205
• shadow Migration: CSR (Cloud Source Repositories) EOL June 2024 - move to SSM (Secure Source Manager) as default CICD repository GoogleCloudPlatform/pbmm-on-gcp-onboarding#439
• reference SSH authentication option over gcloud API clone in Cloud Build module in bootstrap fails on CSR clone gcloud API auth during push-to-repo.sh for CB trigger creation on particular client - use ssh key and ssh-agent GoogleCloudPlatform/pbmm-on-gcp-onboarding#431

Cloud Source Repositories is scheduled for end of sale on June 17, 2024. Starting June 17, 2024, if your organization hasn't previously used Cloud Source Repositories, you cannot enable the API or use Cloud Source Repositories. New projects not connected to an organization can’t enable the Cloud Source Repositories API after June 17, 2024. Customers who have already enabled the API prior to this date will not be affected and can continue to use Cloud Source Repositories.

Terraform Resources

1.3.10

Detailed design

Work is in progress in also bringing in ADO (Azure DevOps) as a CI/CD option - as it is the default repository/pipeline tool for 80% of CA PubSec clients

https://github.com/terraform-google-modules/terraform-example-foundation/issues/1205

Additional information

fmichaelobrien will look into the SSM addition unless this work is already assigned in the roadmap

[obriensystems]

module references

• https://github.com/terraform-google-modules/terraform-google-bootstrap/blob/master/modules/tf_cloudbuild_source/main.tf

[eeaton]

Thanks for raising this issue. This is identified on our internal roadmap as work to address in the next major round of updates for v5 (sometime in H2 this year). The answer may or may not be SSM, due to the limitation that SSM is currently an invitation only service.

[sleighton2022]

Closing this issue, and as eeaton noted, we will address this as part of our H2 roadmap.

[sleighton2022]

Reopening, as I didn't notice it was marked as backlog.

[fmichaelobrien]

Sounds good. I am currently working a PR patch in our fork for later submission.

[eeaton]

One additional aspect to address when we work on this:

It was identified in #1273 that running the docker tests documented in CONTRIBUTING.md also have a dependency on Cloud Build & CSR, so this will fail in any new organizations that haven't previously used CSR. It's not yet clear to me whether we can unpick this locally, or it's an upstream issue with the CFT test framework.

Update: comment from apeabody suggests that this might be the API enablement in the bootstrap project, not necessarily the framework:

Hi @eeaton - I suspect (without seeing diagnostic output) that the make docker_test_prepare dependency on Cloud Source Repositories is due to the sourcerepo API activation in this repo's test/setup: https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/test/setup/main.tf#L65C6-L65C16
That line could likely be commented out to verify or if not using Cloud Source Repositories.

[Marrangas]

As noted SSM is invitation only. Most vcs can be currently integrated with cloud buls 2nd generation repos.

Thanks for raising this issue. This is identified on our internal roadmap as work to address in the next major round of updates for v5 (sometime in H2 this year). The answer may or may not be SSM, due to the limitation that SSM is currently an invitation only service.

Fully automating that cloud build repository connection can be a little fussy. It is possible to expose a veriable vsc_cb2nd_enabled = false to wait until that connection is made manually and then change it, to be able to bootstrap with the current tools and still create the pipelines in cloud build.

[eeaton]

Tracking additional feedback from this thread that directions for deploying locally in 0-boostrap are unclear.

There was an assumption before that the Deploy with Cloud Build directions in 0-bootstrap were the same for deploying locally, however

• that's not clear in the text (deploy locally directions never actually say that)
• the deploy cloud build directions have some dependencies on CSR, so don't work for deploying locally without some understanding of which parts to keep and which to ignore

We'll need to address this as part of reconfiguring the default deployment path and rewriting directions.

[caetano-colin]

Issue is being tracked on the following PR:

#1329