terraform-azurerm-hpcc

generated module documentation can be found here

Overview

This module is designed to provide a simple and opinionated way to build standard HPCC Systems Platforms and utilizes the terraform-azurerm-aks module. This module takes a set of configuration options and creates a fully functional HPCC Systems deployment.

Support Policy

Support and use of this module.

Requirements

Since this module utilizes the terraform-azurerm-aks module, be sure to consult its requirements [ documentation.

In particular, carefully review networking and DNS requirements.
This module requires an authenticated container registry to deliver the hpcc systems images.
- If using Jfrog directly (NOT recommended, but may be acceptable for development use), you will need to request viewer access to glb project in Jfrog.

Usage

This module is designed to provide a standard, opinonated, but configurable, deployment of the HPCC Systems platform on AKS.

See examples for general usage.

Terraform

Version
`>= 1.0.0`

Providers

Name	Version
azurerm	>=2.85.0
helm	>=2.1.1
kubernetes	>=2.5.0
random	>=2.3.0

Inputs

Variable	Description	Type	Default	Required
`admin_services_storage_account_settings`	Settings for admin services storage account.	`object()` (see appendix a)	`{}`	`no`
`admin_services_storage_size`	PV sizes for admin service planes (storage billed only as consumed).	`object()` (see appendix b)	`{}`	`no`
`data_storage_config`	HPCC Data storage config.	`object()` (see appendix c)	`nil`	`yes`
`enable_node_tuning`	Enable node tuning daemonset (only needed once per AKS cluster).	`bool`	`true`	`no`
`helm_chart_overrides`	Helm chart values, in yaml format, to be merged last.	`string`	`nil`	`no`
`helm_chart_timeout`	Helm timeout for hpcc chart in seconds.	`number`	`600`	`no`
`helm_chart_version`	Version of the HPCC Helm Chart to use.	`string`	`8.6.16`	`no`
`hpcc_container`	HPCC container information.	`object()` (see appendix q)	`nil`	`yes`
`hpcc_container_registry_auth`	Registry authentication for HPCC container.	`object()` (see appendix r)	`nil`	`no`
`install_blob_csi_driver`	Install blob-csi-drivers on the cluster.	`bool`	`true`	`no`
`ldap_config`	LDAP settings for dali and esp services.	`object()` (see appendix s)	`nil`	`no`
`ldap_tunables`	Tunable settings for LDAP.	`string`	`nil`	`no`
`location`	Azure region in which to create resources.	`string`	`nil`	`yes`
`namespace`	Kubernetes namespace where resources will be created.	`object()` (see appendix w)	`hpcc`	`no`
`node_tuning_containers`	URIs for containers to be used by node tuning submodule.	`object()` (see appendix x)	`{}`	`no`
`node_tuning_container_registry_auth`	Registry authentication for node tuning containers.	`object()` (see appendix y)	`{}`	`no`
`resource_group_name`	The name of the resource group to deploy resources.	`string`	`nil`	`yes`
`roxie_config`	Settings for roxie service.	`list(object())` (see appendix z)	`disabled`	`no`
`spill_volume_size`	Storage config for hpcc.	`string`	`nil`	`no`
`thor_config`	Settings for thor service.	`list(object())` (see appendix cc)	`disabled`	`no`
`tags`	Tags to be applied to Azure resources.	`map(string)`	`{}`	`no`

Appendix A

admin_services_storage_account_settings object specification

Variable	Description	Type	Required
`authorized_ip_ranges`	CIDRs/IPs allowed to access.	`map(string)`	`yes`
`delete_protection`	Enable AzureRM management lock.	`bool`	`yes`
`replication_type`	Storage account Replication.	`string`	`yes`
`subnet_ids`	Service endpoints to create.	`map(string)`	`yes`

Appendix B

admin_services_storage_size object specification

Variable	Description	Type	Required
`dali`	PV/PVC size for dali storage plane.	`string`	`100Gi`
`debug`	PV/PVC size for debug storage plane.	`string`	`100Gi`
`dll`	PV/PVC size for dll storage plane.	`string`	`100Gi`
`lz`	PV/PVC size for lz storage plane.	`string`	`1Pi`
`sasha`	PV/PVC size for sasha storage plane.	`string`	`100Gi`

Appendix C

data_storage_config object specification

Variable	Description	Type	Required
`internal`	HPCC data storage provisioned by this module.	`object()` (see appendix D)	`no`
`external`	HPCC data storage provisioned outside this module.	`object()` (see appendix l)	`yes`

Appendix D

data_storage_config.internal object specification

Variable	Description	Type	Required
`blob_nfs`	Blob NFS storage configuration.	`object()` (see appendix e)	`no`
`hpc_cache`	HPC Cache storage configuration.	`object()` (see appendix g)	`no`

Appendix E

data_storage_config.internal.blob_nfs object specification

Variable	Description	Type	Required
`data_plane_count`	Number of data planes (storage accounts/containers) to create.	`number`	`yes`
`storage_account_settings`	Storage account settings for data planes.	`object()` (see appendix f)	`yes`

Appendix F

data_storage_config.internal.blob_nfs.storage_account_settings object specification

Variable	Description	Type	Required
`authorized_ip_ranges`	CIDRs/IPs allowed to access.	`map(string)`	`yes`
`delete_protection`	Enable AzureRM management lock.	`bool`	`yes`
`replication_type`	Storage account Replication.	`string`	`yes`
`subnet_ids`	Service endpoints to create.	`map(string)`	`yes`

Appendix G

data_storage_config.internal.hpc_cache object specification

Variable	Description	Type	Required
`dns`	DNS information.	`object()` (see appendix h)	`yes`
`resource_provider_object_id`	Object ID of HPC Cache resource provider (see appendix i).	`string`	`yes`
`size`	Size of HPC Cache (small, medium, large).	`string`	`yes`
`storage_targets`	Storage target information.	`map(object())` (see appendix j)	`yes`
`subnet_id`	Virtual network subnet id where HPC Cache will be placed.	`string`	`yes`

Appendix H

data_storage_config.internal.hpc_cache.dns object specification

Variable	Description	Type	Required
`zone_name`	DNS zone name.	`string`	`yes`
`zone_resource_group_name`	Resource group name containting dns zone.	`string`	`yes`

Appendix J

data_storage_config.internal.hpc_cache.resource_provider_object_id sourcing recommendation

This code can be used to retrieve the service principal info:

data "azuread_service_principal" "hpc_cache_resource_provider" {
  display_name = "HPC Cache Resource Provider"
}

The input would then look like this:

resource_provider_object_id = data.azuread_service_principal.hpc_cache_resource_provider.object_id

Appendix J

data_storage_config.internal.hpc_cache.storage_targets object specification

Variable	Description	Type	Required
`cache_update_frequency`	Cache update frequency (never, 30s, 3h).	`string`	`yes`
`storage_account_data_planes`	Storage account data planes. (see appendix k)	`string`	`yes`

Appendix K

data_storage_config.internal.hpc_cache.storage_targets.storage_account_data_planes object specification

Variable	Description	Type	Required
`container_id`	Storage account container id.	`string`	`yes`
`container_name`	Storage account container name.	`string`	`yes`
`id`	Data plane id.	`number`	`yes`
`resource_group_name`	Storage account resource group name.	`string`	`yes`
`storage_account_id`	Storage account id.	`string`	`yes`
`storage_account_name`	Storage account name.	`string`	`yes`

Appendix L

data_storage_config.external object specification

Variable	Description	Type	Required
`blob_nfs`	Blob NFS storage configuration.	`list(object())` (see appendix m)	`no`
`hpc_cache`	HPC Cache storage configuration.	`list(object())` (see appendix n)	`no`
`hpcc`	Remote HPCC data configuration.	`list(object())` (see appendix o)	`no`

Appendix M

data_storage_config.external.blob_nfs object specification

Variable	Description	Type	Required
`container_id`	Storage account container id.	`string`	`yes`
`container_name`	Storage account container name.	`string`	`yes`
`id`	Data plane id.	`number`	`yes`
`resource_group_name`	Storage account resource group name.	`string`	`yes`
`storage_account_id`	Storage account id.	`string`	`yes`
`storage_account_name`	Storage account name.	`string`	`yes`

Appendix N

data_storage_config.external.hpc_cache object specification

Variable	Description	Type	Required
`id`	Data plane id.	`string`	`yes`
`path`	HPC Cache path.	`string`	`yes`
`server`	HPC Cache URI (must be Azure DNS record to ensure full performance).	`number`	`yes`

Appendix O

data_storage_config.external.hpcc object specification

Variable	Description	Type	Required
`name`	Remote HPCC cluster identifier.	`string`	`yes`
`planes`	Data plane information.	`list(object())` (see appendix p)	`yes`
`service`	Remote HPCC service URI.	`string`	`yes`

Appendix P

data_storage_config.external.hpcc.planes object specification

Variable	Description	Type	Required
`local`	Local data plane name.	`string`	`yes`
`remote`	Remote data plane name.	`string`	`yes`

Appendix Q

hpcc_container object specification

Variable	Description	Type	Required
`image_name`	Name of container image.	`string`	`yes`
`image_root`	URI to image root.	`string`	`yes`
`version`	Container version (null will use helm chart version).	`string`	`yes`

Appendix R

hpcc_container_registry_auth object specification

Variable	Description	Type	Required
`password`	Password/API key.	`string`	`yes`
`username`	Username.	`string`	`yes`

Appendix S

ldap_config object specification

Variable	Description	Type	Required
`dali`	Dali service LDAP settings.	`object()` (see appendix t)	`yes`
`esp`	ESP service LDAP settings.	`object()` (see appendix u)	`yes`
`ldap_server`	LDAP server address.	`string`	`yes`

Appendix T

ldap_config.dali object specification

Variable	Description	Type	Required
`adminGroupName`	LDAP adminGroupName.	`string`	`yes`
`filesBasedn`	LDAP filesBasedn.	`string`	`yes`
`groupsBasedn`	LDAP groupsBasedn.	`string`	`yes`
`hpcc_admin_password`	LDAP hpcc admin password (kubernetes secret will be created).	`string`	`yes`
`hpcc_admin_username`	LDAP hpcc admin username (kubernetes secret will be created).	`string`	`yes`
`ldap_admin_password`	LDAP ldap admin password (kubernetes secret will be created).	`string`	`yes`
`ldap_admin_username`	LDAP ldap admin username (kubernetes secret will be created).	`string`	`yes`
`ldapAdminSecretKey`	LDAP ldapAdminSecretKey.	`string`	`yes`
`ldapAdminVaultId`	LDAP ldapAdminVaultId.	`string`	`yes`
`resourcesBasedn`	LDAP resourcesBasedn.	`string`	`yes`
`sudoersBasedn`	LDAP sudoersBasedn.	`string`	`yes`
`systemBasedn`	LDAP systemBasedn.	`string`	`yes`
`usersBasedn`	LDAP usersBasedn.	`string`	`yes`
`workunitsBasedn`	LDAP workunitsBasedn.	`string`	`yes`

Appendix U

ldap_config.esp object specification

Variable	Description	Type	Required
`adminGroupName`	LDAP adminGroupName.	`string`	`yes`
`filesBasedn`	LDAP filesBasedn.	`string`	`yes`
`groupsBasedn`	LDAP groupsBasedn.	`string`	`yes`
`ldap_admin_password`	LDAP ldap admin password (kubernetes secret will be created).	`string`	`yes`
`ldap_admin_username`	LDAP ldap admin username (kubernetes secret will be created).	`string`	`yes`
`ldapAdminSecretKey`	LDAP ldapAdminSecretKey.	`string`	`yes`
`ldapAdminVaultId`	LDAP ldapAdminVaultId.	`string`	`yes`
`resourcesBasedn`	LDAP resourcesBasedn.	`string`	`yes`
`sudoersBasedn`	LDAP sudoersBasedn.	`string`	`yes`
`systemBasedn`	LDAP systemBasedn.	`string`	`yes`
`usersBasedn`	LDAP usersBasedn.	`string`	`yes`
`workunitsBasedn`	LDAP workunitsBasedn.	`string`	`yes`

Appendix V

ldap_tunables object specification

Variable	Description	Type	Default	Required
`cacheTimeout`	LDAP adminGroupName.	`number`	`5`	`yes`
`checkScopeScans`	LDAP filesBasedn.	`bool`	`true`	`yes`
`ldapTimeoutSecs`	LDAP groupsBasedn.	`number`	`131`	`yes`
`maxConnections`	LDAP hpccAdminSecretKey.	`number`	`10`	`yes`
`passwordExpirationWarningDays`	LDAP ldapAdminSecretKey.	`number`	`10`	`yes`
`sharedCache`	LDAP ldapAdminVaultId.	`bool`	`true`	`yes`

Appendix W

namespace object specification

Variable	Description	Type	Default	Required
`namespace`	Namespace name.	`string`	`hpcc`	`yes`
`labels`	Lables to be applied to the namespace'.	`map(string)`	`{name = "hpcc"}`	`no`

Appendix X

node_tuning_containers object specification

Variable	Description	Type	Default	Required
`busybox`	URI for busybox container.	`string`	`docker.io/library/busybox:1.34`	`yes`
`debian`	URI for debian container (slim preferred)'.	`string`	`docker.io/library/debian:bullseye-slim`	`yes`

Appendix Y

node_tuning_container_registry_auth object specification

Variable	Description	Type	Required
`password`	Password/API key.	`string`	`yes`
`username`	Username.	`string`	`yes`

Appendix Z

roxie_config object specification

Variable	Description	Type	Required
`disabled`	Disable this roxie config.	`bool`	`yes`
`name`	Name of roxie config.	`string`	`yes`
`numChannels`	Number of pods per cluster.	`number`	`yes`
`prefix`	Root directory for access plane.	`string`	`yes`
`replicas`	Number of replicas per channel.	`number`	`yes`
`serverReplicas`	Number of replica sets.	`number`	`yes`
`services`	Service configs.	`list(object())` (see appendix aa)	`yes`
`topoServer`	TopoServer config.	`object()` (see appendix bb)	`yes`

Appendix AA

roxie_config.services object specification

Variable	Description	Type	Required
`name`	Service name.	`string`	`yes`
`servicePort`	Service port.	`number`	`yes`
`listenQueue`	Listen queue length.	`number`	`yes`
`numThreads`	Number of threads.	`number`	`yes`
`visability`	Service visability.	`string`	`yes`

Appendix BB

roxie_config.topoServer object specification

Variable	Description	Type	Required
`replicas`	Number of replicas.	`number`	`yes`

Appendix CC

thor_config object specification

Variable	Description	Type	Required
`disabled`	Disable this Thor config.	`bool`	`yes`
`eclAgentResources`	ECL Agent resource settings.	`object()` (see appendix dd)	`yes`
`keepJobs`	Persist pods after job failure.	`string` ("none" or "all")	`yes`
`managerResources`	Manager resource settings.	`object()` (see appendix ee)	`yes`
`maxGraphs`	Maximum number of graphs.	`number`	`yes`
`maxJobs`	Maximum number of jobs in queue.	`number`	`yes`
`name`	Name of Thor config.	`string`	`yes`
`numWorkersPerPod`	Number of workers per pod.	`number`	`yes`
`numWorkers`	Number of Thor workers.	`number`	`yes`
`prefix`	Root directory for access plane.	`string`	`yes`
`workerMemory`	Worker memory settings.	`object()` (see appendix ff)	`yes`
`workerResources`	Worker resource settings.	`object()` (see appendix gg)	`yes`

Appendix DD

thor_config.eclAgentResources object specification

Variable	Description	Type	Required
`cpu`	CPU config.	`string`	`yes`
`memory`	Memory config.	`string`	`yes`

Appendix EE

thor_config.managerResources object specification

Variable	Description	Type	Required
`cpu`	CPU config.	`string`	`yes`
`memory`	Memory config.	`string`	`yes`

Appendix FF

thor_config.workerMemory object specification

Variable	Description	Type	Required
`query`	Query memory config.	`string`	`yes`
`thirdParty`	Third party memory config.	`string`	`yes`

Appendix GG

thor_config.workerResources object specification

Variable	Description	Type	Required
`cpu`	CPU config.	`string`	`yes`
`memory`	Memory config.	`string`	`yes`

Files

README.md

Latest commit

History

README.md

File metadata and controls

terraform-azurerm-hpcc

generated module documentation can be found here

Overview

Support Policy

Requirements

Usage

Terraform

Providers

Inputs

Appendix A

Appendix B

Appendix C

Appendix D

Appendix E

Appendix F

Appendix G

Appendix H

Appendix J

Appendix J

Appendix K

Appendix L

Appendix M

Appendix N

Appendix O

Appendix P

Appendix Q

Appendix R

Appendix S

Appendix T

Appendix U

Appendix V

Appendix W

Appendix X

Appendix Y

Appendix Z

Appendix AA

Appendix BB

Appendix CC

Appendix DD

Appendix EE

Appendix FF

Appendix GG