The main purpose of this repository is to terraform all the resources needed for Exam AI-102: Designing and Implementing a Microsoft Azure AI Solution - Certification.
The end goal is to be easily deploy all the resources needed for the self-paced learning modules. As I have created resources following the instructions in the lab exercises when using the Azure Portal UI. Please note your your mileage may (or might) vary, as these resources were deployed using my personal account which has no restrictions.
Table of contents
- terraform
- terragrunt
- terraform-docs this is required for
terraform_docshooks - pre-commit
- Have a Azure Portal account.
- You will need to create a Service Principal with a Client Secret follow instructions.
- Optionally link storage account created for cognitive services
- Create scripts that will give the search services the correct Azure IAM roles via API calls
- Move the creation of storage accounts into separate module
- Terraform creation of Azure Kubernetes Service for machine learning
- Navigate to the environment you would like to deploy,
- Plan your changes with
terragrunt planto see what changes will be made, - If you're happy with the changes
terragrunt apply.
IMPORTANT
Please note that
.tfstatefiles are stored locally on your machine on first apply, an Azure Storage account is created as part of the Terraform. However you will be required to migrate to it after the tfstate storage account has been created. Please see comments inbackend.tfor any of the environmentterragrunt.hcl.
A majority of the resources created will have either the 'Standard' or 'Free' tier used, however this does not mean that it will be cheap. Please be mindful of the cost for each tier, for example the Azure Container Instance is always running and you will be charged for it's up-time during the month. Infracost has been used to help indicate how much it will cost you to have all these resources created.
Predicted Infracost as of 24/08/2023
Name Monthly Qty Unit Monthly Cost
azurerm_key_vault_key.tfstate_key_vault_key
├─ Secrets operations Monthly cost depends on usage: $0.03 per 10K transactions
├─ Storage key rotations Monthly cost depends on usage: $1.00 per renewals
└─ Software-protected keys Monthly cost depends on usage: $0.03 per 10K transactions
azurerm_log_analytics_workspace.tfstate_analytics_workspace
├─ Log data ingestion Monthly cost depends on usage: $2.99 per GB
├─ Log data export Monthly cost depends on usage: $0.13 per GB
├─ Basic log data ingestion Monthly cost depends on usage: $0.65 per GB
├─ Basic log search queries Monthly cost depends on usage: $0.0065 per GB searched
├─ Archive data Monthly cost depends on usage: $0.026 per GB
├─ Archive data restored Monthly cost depends on usage: $0.13 per GB
└─ Archive data searched Monthly cost depends on usage: $0.0065 per GB
azurerm_search_service.cognitive_search_service
├─ Search usage (Basic, 1 unit) 730 hours $73.73
└─ Image extraction (first 1M) Monthly cost depends on usage: $1.00 per 1000 images
azurerm_storage_account.tfstate
├─ Capacity Monthly cost depends on usage: $0.0392 per GB
├─ Write operations Monthly cost depends on usage: $0.11 per 10k operations
├─ List and create container operations Monthly cost depends on usage: $0.11 per 10k operations
├─ Read operations Monthly cost depends on usage: $0.0043 per 10k operations
├─ All other operations Monthly cost depends on usage: $0.0043 per 10k operations
└─ Blob index Monthly cost depends on usage: $0.075 per 10k tags
module.cognitive_services.azurerm_storage_account.cognitive_service_storage[0]
├─ Capacity Monthly cost depends on usage: $0.0392 per GB
├─ Write operations Monthly cost depends on usage: $0.11 per 10k operations
├─ List and create container operations Monthly cost depends on usage: $0.11 per 10k operations
├─ Read operations Monthly cost depends on usage: $0.0043 per 10k operations
├─ All other operations Monthly cost depends on usage: $0.0043 per 10k operations
└─ Blob index Monthly cost depends on usage: $0.075 per 10k tags
module.form_recognizer.azurerm_storage_account.cognitive_service_storage[0]
├─ Capacity Monthly cost depends on usage: $0.0392 per GB
├─ Write operations Monthly cost depends on usage: $0.11 per 10k operations
├─ List and create container operations Monthly cost depends on usage: $0.11 per 10k operations
├─ Read operations Monthly cost depends on usage: $0.0043 per 10k operations
├─ All other operations Monthly cost depends on usage: $0.0043 per 10k operations
└─ Blob index Monthly cost depends on usage: $0.075 per 10k tags
module.machine_learning.azurerm_application_insights.machine_learning_key_insights
└─ Data ingested Monthly cost depends on usage: $2.30 per GB
module.machine_learning.azurerm_container_registry.machine_learning_container_registry
├─ Registry usage (Basic) 30 days $5.00
├─ Storage (over 10GB) Monthly cost depends on usage: $0.10 per GB
└─ Build vCPU Monthly cost depends on usage: $0.0001 per seconds
module.machine_learning.azurerm_storage_account.machine_learning_storage
├─ Capacity Monthly cost depends on usage: $0.0392 per GB
├─ Write operations Monthly cost depends on usage: $0.11 per 10k operations
├─ List and create container operations Monthly cost depends on usage: $0.11 per 10k operations
├─ Read operations Monthly cost depends on usage: $0.0043 per 10k operations
├─ All other operations Monthly cost depends on usage: $0.0043 per 10k operations
└─ Blob index Monthly cost depends on usage: $0.075 per 10k tags
module.video_indexer_media_services.azurerm_storage_account.media_storage
├─ Capacity Monthly cost depends on usage: $0.0392 per GB
├─ Write operations Monthly cost depends on usage: $0.11 per 10k operations
├─ List and create container operations Monthly cost depends on usage: $0.11 per 10k operations
├─ Read operations Monthly cost depends on usage: $0.0043 per 10k operations
├─ All other operations Monthly cost depends on usage: $0.0043 per 10k operations
└─ Blob index Monthly cost depends on usage: $0.075 per 10k tags
OVERALL TOTAL $78.73
──────────────────────────────────
64 cloud resources were detected:
∙ 11 were estimated, all of which include usage-based costs, see https://infracost.io/usage-file
∙ 41 were free:
∙ 14 x azurerm_key_vault_secret
∙ 9 x azurerm_key_vault
∙ 7 x azurerm_monitor_activity_log_alert
∙ 3 x azurerm_storage_container
∙ 1 x azurerm_key_vault_access_policy
∙ 1 x azurerm_log_analytics_storage_insights
∙ 1 x azurerm_resource_group
∙ 1 x azurerm_role_assignment
∙ 1 x azurerm_search_service
∙ 1 x azurerm_storage_account_customer_managed_key
∙ 1 x azurerm_storage_account_network_rules
∙ 1 x azurerm_user_assigned_identity
∙ 12 are not supported yet, see https://infracost.io/requested-resources:
∙ 7 x azurerm_cognitive_account
∙ 1 x azurerm_container_group
∙ 1 x azurerm_machine_learning_compute_cluster
∙ 1 x azurerm_machine_learning_workspace
∙ 1 x azurerm_media_services_account
∙ 1 x azurerm_resource_group_template_deploymentGit hook scripts are very helpful for identifying simple issues before pushing any changes. Hooks will run on every commit automatically pointing out issues in the code e.g. trailing whitespace.
To help with the maintenance of these hooks, pre-commit is used, along with pre-commit-hooks.
Please following these instructions to install pre-commit locally and ensure that you have run pre-commit install to install the hooks for this project.
Additionally, once installed, the hooks can be updated to the latest available version with pre-commit autoupdate.
Code formatting and documentation for variables and outputs is generated using pre-commit-terraform hooks that in turn uses terraform-docs that will insert/update documentation. The following markers have been added to the README.md:
<!-- {BEGIN|END}_TF_DOCS --->
| Name | Version |
|---|---|
| terraform | = 1.5.4 |
| azurerm | 3.67.0 |
| random | 3.5.1 |
| Name | Version |
|---|---|
| azurerm | 3.67.0 |
| random | 3.5.1 |
| Name | Source | Version |
|---|---|---|
| cognitive_services | ./modules/cognitive_services | n/a |
| cognitive_services_container_language | ./modules/container_instances | n/a |
| custom_question_answer_service | ./modules/cognitive_services | n/a |
| custom_vision_service_prediction | ./modules/cognitive_services | n/a |
| custom_vision_service_training | ./modules/cognitive_services | n/a |
| form_recognizer | ./modules/cognitive_services | n/a |
| language_service | ./modules/cognitive_services | n/a |
| machine_learning | ./modules/machine_learning | n/a |
| open_ai | ./modules/cognitive_services | n/a |
| video_indexer_media_services | ./modules/video_indexers | n/a |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| arm_client_id | The Client ID which should be used. This can also be sourced from the ARM_CLIENT_ID Environment Variable. |
string |
n/a | yes |
| arm_client_secret | The Client Secret which should be used. This can also be sourced from the ARM_CLIENT_SECRET Environment Variable. |
string |
n/a | yes |
| arm_subscription_id | The Subscription ID which should be used. This can also be sourced from the ARM_SUBSCRIPTION_ID Environment Variable. |
string |
n/a | yes |
| arm_tenant_id | The Tenant ID which should be used. This can also be sourced from the ARM_TENANT_ID Environment Variable. |
string |
n/a | yes |
| cloud_enviornment | The Cloud Environment which should be used. Possible values are public,usgovernment, german, and china. Defaults to public. This can also be sourced from the ARM_ENVIRONMENT Environment Variable. |
string |
"public" |
no |
| environment | The name of the environment to help identify resources. | string |
n/a | yes |
| location | The Azure Region where the Resource Group should exist. Changing this forces a new Resource Group to be created. |
string |
"West Europe" |
no |
| personal_ip_address | Add your client IP address to the networking to allow access. | string |
n/a | yes |
| tags | Tags to be added to resources created. | map(string) |
{} |
no |
| Name | Description |
|---|---|
| cognitive_service_endpoint | The endpoint used to connect to the Cognitive Service Account. |
| cognitive_service_key_vault_name | The name of the key vault created to contain cognitive service secrets. |
| cognitive_service_primary_access_key | The primary access key which can be used to connect to the Cognitive Service Account. |
| cognitive_service_secondary_access_key | The secondary access key which can be used to connect to the Cognitive Service Account. |
| cognitive_services_container_language_fdqn | The FDQN to connect to the container instance. |
| custom_vision_service_prediction_endpoint | The endpoint used to connect to the custom vision prediction service Account. |
| custom_vision_service_prediction_key_vault_name | The name of the key vault created to contain custom vision secrets. |
| custom_vision_service_prediction_primary_access_key | The primary access key which can be used to connect to the Cognitive Service Account. |
| custom_vision_service_prediction_secondary_access_key | The secondary access key which can be used to connect to the Cognitive Service Account. |
| custom_vision_service_training_endpoint | The endpoint used to connect to the custom vision training service Account. |
| custom_vision_service_training_key_vault_name | The name of the key vault created to contain custom vision secrets. |
| custom_vision_service_training_primary_access_key | The primary access key which can be used to connect to the Cognitive Service Account. |
| custom_vision_service_training_secondary_access_key | The secondary access key which can be used to connect to the Cognitive Service Account. |
| form_recognizer_endpoint | The endpoint used to connect to the form recognizer Account. |
| form_recognizer_key_vault_name | The name of the key vault created to contain form recognizer secrets. |
| form_recognizer_primary_access_key | The primary access key which can be used to connect to the Cognitive Service Account. |
| form_recognizer_secondary_access_key | The secondary access key which can be used to connect to the Cognitive Service Account. |
| language_service_endpoint | The endpoint used to connect to the Language Service Account. |
| language_service_key_vault_name | The name of the key vault created to contain language service secrets. |
| language_service_primary_access_key | The primary access key which can be used to connect to the Language Service Account. |
| language_service_secondary_access_key | The secondary access key which can be used to connect to the Language Service Account. |
| machine_learning_discovery_url | The url for the discovery service to identify regional endpoints for machine learning experimentation services. |
| machine_learning_workspace_id | The immutable id associated with this workspace. |
| service_principal_client_id | The principal being used to apply terraform changes for this subscription. |
| tenant_id | The tenant ID used for this subscription. |
| tfstate_resource_group_name | The name of the resource group created for the Terraform tfstate. |
| tfstate_storage_account_key | The storage account key created for the Terraform tfstate. |
| tfstate_storage_account_name | The name of the storage account created for the Terraform tfstate. |
| tfstate_storage_container_name | The name of the storage container created for the Terraform tfstate. |