Implement tag-based deployment model

[thekaveman]

Corollary to cal-itp/benefits#1527

We may want to try this out here in Server first, where it is less likely to cause a disruption to the dev flow etc.

---

WIP - Rough sketch of how we could implement this:

What we want:

Have 1 branch called main.

Deployments to dev environment

• Opening a PR targeting main triggers Azure DevOps pipeline to run terraform plan
• Merging an approved PR into main triggers Azure DevOps pipeline to run terraform apply
• Merging an approved PR into main triggers GH Action to push tagged Docker image to GHCR
  • Azure uses a webhook to listen for Docker image push and restarts the app service

Deployments to test and prod environments

• Pushing a Git tag with the release naming pattern an -rcX postfix triggers Azure DevOps to run for test - terraform plan runs first, manual approval is needed, and then terraform apply runs
• Pushing a Git tag with the release naming pattern and without the -rcX postfix triggers Azure DevOps to run for prod - terraform plan runs first, manual approval is needed, and then terraform apply runs. This also creates a GitHub release for the tag.
• Azure uses a webhook to listen for Docker image push and restarts the app service

Changes needed:

Most of the changes we need are for test/prod deployment.

• Add support for running infra pipeline when Git tag is pushed with specific name #284
• Add support for running "Deploy" GitHub workflow when GitHub pre-release/release is created #286
• Add support for app getting its version dynamically #287
• Automate creation of GitHub release when Git tag is pushed with specific name #393
• Do a release with new deployment model #288
• Fully migrate to new deployment model #290
• Update documentation to reflect new deployment model #289

Needs further research:

• GitHub permissions
• How the app gets its version setuptools_scm
• A question that popped up while writing all this: based on the current pipeline YAML, do we run terraform apply for the test workspace when a PR is opened against the test branch? That doesn't seem right - I would've said it should run a terraform plan. (This doesn't affect new deployment model, but might be a benign bug in the current deployment model?) Figured out that this is not a bug -- see Investigating infra pipeline logic #285

[thekaveman]

@angela-tran will start scoping this out for Eligibility Server.

Feel free to edit the original issue description with any details/lists/etc.

[angela-tran]

I've written up a WIP plan in the issue description for how we could implement the idea. Still thinking through all this

[thekaveman]

WIP - Rough sketch of how we could implement this

This is great so far. Couple of quick thoughts:

Add logic to derive Docker tag name from Git tag instead of Git branch for test/prod

The app service web hooks are watching for specific tags dev, test, and prod right? Or maybe this is Benefits / the old way we did it there that I'm thinking of.

GitHub permissions

Mentioned this in cal-itp/benefits#1527 too but let's look at tag protections.

How the app gets its version

I have a possible solution for this, I've been exploring in another project https://setuptools-git-versioning.readthedocs.io/en/stable/install.html#pyproject-toml

[angela-tran]

Add logic to derive Docker tag name from Git tag instead of Git branch for test/prod

The app service web hooks are watching for specific tags dev, test, and prod right? Or maybe this is Benefits / the old way we did it there that I'm thinking of.

Oh, yep that's right. The app service is looking for a docker_image_tag that matches the environment name. I'll update the checklist by adding a bullet point to say we need to tag the main branch's Docker image as dev.

[angela-tran]

Adding some clarification from our discussion earlier today:

• we'll need dev, test, and prod Docker tags
• we also want to have Docker tags named with the Git tag name

The plan in the issue description was updated to reflect this.

[angela-tran]

@thekaveman Do you think the checkboxes in the issue description are ready to be turned into issues to be worked on? I added in your thoughts about looking into tag protections and about using setuptools-git-versioning, but I'm not sure if it's ok that the research on those would be done as a part of the issues to be created.