Consider a "Deploy on Release" model

[thekaveman]

Today, our release process requires a number of manual steps, and at least 1 Pull Request requiring interaction from at least 2 people (the author and an approver). This works well but is cumbersome, and could be more efficient especially for getting urgent or timely changes out.

Further, we have 3 separate "versions" of the code across the dev, test, and prod branches. The explicit branch naming makes it easy to see where each environment is, but it also means we're constantly merging and needing to remember the state of each branch (e.g. is test ready to release to prod? Did we merge that fix into test yet?)

The final step in our release process today is to tag and create a GitHub release.

I propose we invert this model and instead make the tagging and GitHub release the initial step that kicks off the rest of the deployment.

Proposal

Branches

• We remove the 3 environment branches: dev, test, and prod
• All new code is merged to a single default branch main
• dev would essentially be renamed to main but otherwise function the same

Deployments

• Merging to main deploys to the dev environment (just like merging to dev today)
• Creating a GitHub pre-release deploys to the test environment
• Create a GitHub release deploys to the prod environment

Discussion

Creating a GitHub Release requires an associated git tag.

Today we tag the prod branch at the end of the process, just before creating the GitHub Release.

Under this new proposed model:

• All the code (generally) is in a single branch / timeline main
• We tag at the beginning of the process to signal a release
• We have far more tags on the main branch, as both test and prod releases are tagged
• Releases are just specific points in the timeline ("forward" or "backward")
• We could think about using tag names or partial names to signify specific features (e.g. sacrt or veterans)
• release == tag == commit

Pros

• Simpler release process: choose a commit, tag it, push the tag, make a (pre) release in GitHub
• Reduced friction to release: no more PRs for approval, drastically reduced release checklist
• Easier to get features and fixes out sooner
• Features and fixes can be released to test / prod independent of where or whether the commit is in main

Cons

• There will be a lot of tags
• It may be too easy to make a deployment? E.g. broken code, tests not passing
• Tagging/releasing from multiple branches, forward and backward in the timeline, may be even more overhead than our 3 straightforward branches

Open questions

• Are existing branch protections for who can write to main enough? Look at the beta tag protections?
  • Who can add tags and make releases (GitHub permissions)?
• Do we want to support deployments from other branches besides main?
• How will we handle updating the app's version? Can't update a file manually anymore, get it from git tags/sha?
• How can we ensure our checks have run on the commit to be deployed? e.g. pytest, cypress, pre-commit, codeql etc.

Additional context

This model is typical of how we deploy Python packages, e.g. eligibility-api.

This model gets us closer to going full CI/CD #1253

[angela-tran]

It may be too easy to make a deployment? E.g. broken code, tests not passing

In our discussion today, we noted we could run the tests for a commit that is tagged for deployment.

If we want to continue requiring approvals for deployments, we could make use of the "Required reviewers" setting on GitHub Environments. See:

• https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers
• https://cloudlumberjack.com/posts/github-actions-approvals/

[angela-tran]

As a part of Backlog review, we looked at this and are thinking this is a good time to do this (will be working on admin interface mostly)

[angela-tran]

Engineers will talk about the plan for this ahead of next sprint planning