Add support for running infra pipeline when Git tag is pushed with specific name

[angela-tran]

• Decide on naming pattern for Git tags for test and prod environments

Do in isolated testing environment

• Update azure-pipeline.yml to be triggered by Git tag for specific naming patterns
• Update logic that decides when to use test and prod/default Terraform workspaces (logic is in workspace.py)
• Update azure-pipeline.yml to require approval for stage that runs terraform apply for test/prod deployment
  • Split up pipeline into multiple stages to allow for this
  • Get terraform plan stage to save a tfplan for the apply command to use
• Make sure the stage that runs terraform plan always runs for test/prod deployment already works as written
• Make sure the stage that runs terraform apply runs for tag-based test/prod deployment
• Code cleanup: make the two stages share variables

Done in #307

Make changes to real pipeline

• Update azure-pipeline.yml to be triggered by Git tag for specific naming patterns
• Update logic that decides when to use test and prod/default Terraform workspaces (logic is in workspace.py)
• Update azure-pipeline.yml to require approval for stage that runs terraform apply for test/prod deployment
• Make sure the stage that runs terraform plan always runs for test/prod deployment already works as written
• Make sure the stage that runs terraform apply runs for tag-based test/prod deployment

Done in #310

[angela-tran]

• Update azure-pipeline.yml to be triggered by Git tag for specific naming patterns

Based on Azure DevOps documentation, this can be done as a "branch" trigger that starts with refs/tags or as a tag trigger. There is a note under the Tags section that makes me think we might want to use the branch trigger. The note implies that a branch matching the tag filter could trigger the pipeline (that's how I'm reading it at least), and we wouldn't want that.

Important

If you specify tags in combination with branch filters, the trigger will fire if either the branch filter is satisfied or the tag filter is satisfied. For example, if a pushed tag satisfies the branch filter, the pipeline triggers even if the tag is excluded by the tag filter, because the push satisfied the branch filter.

[angela-tran]

The note implies that a branch matching the tag filter could trigger the pipeline (that's how I'm reading it at least)

I was reading it incorrectly. After trying out various pipeline runs with various branch/tag names, I understand now that the note was saying: if you tag a commit that is also the commit referenced by a branch, then if the branch name happens to satisfy a branch filter, pushing that tag would trigger the pipeline.

So this means using tag in the triggers section is fine 🙌

[angela-tran]

Updates:

• I've been experimenting with pipeline changes on the branch in Feat: figure out infra pipeline tag trigger by experimenting #307

• As shared during Monday's dev standup, I noticed that my commits to that branch were triggering both the real MST DevOps pipeline as well as another DevOps pipeline I created in our separate compilerllc organization. This doesn't match the documentation that Kegan linked to in Trigger azure-pipelines.yml via GitHub Action #294 which says that only the "first" organization's pipeline will be triggered by commits or pull requests.

• In an effort to isolate my experiments to just a pipeline in compilerllc:

  • I created a copy of the pipeline YAML file called experiment-pipeline.yml at the root of the project
  • Updated its path filter to experiment-* so that it'll only look at my file
  • Reverted the original pipeline YAML back

We installed the Terraform extension in compilerllc so in theory we're now ready to run Terraform tasks. I ran into a blocker just now though, where my pipeline is triggered but then says

No hosted parallelism has been purchased or granted. To request a free parallelism grant, please fill out the following form https://aka.ms/azpipelines-parallelism-request

Googling this error, I see a StackOverflow answer from July 2021 that basically says Azure made a change so that free accounts have to make a request in order to be able to run jobs.

I'm confused by this because I don't remember us having to do that for MST's pipeline, and we created that in November 2022...

See the "Parallel jobs" settings for compilerllc vs. MST's:

Compiler (0 jobs for Microsoft-hosted)	MST (1 parallel job for Microsoft-hosted)

I filled out the request form, and it said "it could take 2-3 business days to proceed the request."

New plan

I'm just gonna change the pipeline in my branch to not actually run Terraform and have it run in MST's DevOps org. This will still allow me to test / figure out the correct syntax for all the tag stuff we want to add to the pipeline while ensuring my experiments don't touch anything in MST's Azure environment.

[angela-tran]

Our request was approved on 8/23/2023, and I've been able to run my pipeline in compilerllc yesterday and today. I replaced the terraform commands with simple echo statements to make testing easier (see a recent pipeline run).

[angela-tran]

@thekaveman My next step is to put back the terraform commands and try to get the terraform plan stage to save a plan file that is then used by the terraform apply stage. This is so we can have an approval check before the apply stage.

To work on this though, I need:

• a service connection between Azure DevOps and the Azure Portal
• an Azure subscription in the "Compiler LLC" directory

Once I have those, I can define a small Terraform file that just creates a Resource Group for my pipeline to run so that it isn't trying to create all the infrastructure for a new eligibility-server.

Should I go ahead and set up the subscription, or do you want to do that? The Pay-As-You-Go option requires entering payment information.

[thekaveman]

@angela-tran Go for it! You can use your card for now. LMK if you get stuck and I can set it up.

[angela-tran]

Things that have to be set up manually in Azure and were done yesterday/today:

• Subscription
• Resource Group for holding Terraform state (backendResourceGroupName is needed in pipeline YAML file and in Terraform file that defines the resource provider)
• Storage Account in that Resource Group
• Container in that Storage Account
• Terraform workspace

[angela-tran]

Something weird I've noticed yesterday and today:

There is sometimes a long delay between pushing the tag and the pipeline getting triggered.

Earlier I did the following:

• created some commits in my branch locally (did not push to GitHub)
• created an annotated tag called 2023.08.202-rc1 on the last commit and pushed to GitHub at 11:15am CT
• created an annotated tag called 2023.08.202 on the last commit and pushed to GitHub at 11:17amCT
• pushed the commits to the remote branch on GitHub, which is associated with a pull request (Feat: figure out infra pipeline tag trigger by experimenting #307)

The pipeline triggered at 11:20am CT for the new commits on the pull request, then at 11:21am CT for the 2023.08.202 tag, and finally at 11:30am for the 2023.08.202-rc1 tag.