Skip to content

This is a copy of my repository on my vm2 where I've made changed to godji's opinionated-hpcc terraform.

Notifications You must be signed in to change notification settings

hpccsystems-solutions-lab/tlh-opinionated-terraform-azurerm-hpcc

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

terraform-azurerm-hpcc

generated module documentation can be found here

Overview

This module is designed to provide a simple and opinionated way to build standard HPCC Systems Platforms and utilizes the terraform-azurerm-aks module. This module takes a set of configuration options and creates a fully functional HPCC Systems deployment.


Support Policy

Support and use of this module.


Requirements

  1. Since this module utilizes the terraform-azurerm-aks module, be sure to consult its requirements [ documentation.

    In particular, carefully review networking and DNS requirements.

  2. This module requires an authenticated container registry to deliver the hpcc systems images.

    • If using Jfrog directly (NOT recommended, but may be acceptable for development use), you will need to request viewer access to glb project in Jfrog.

Usage

This module is designed to provide a standard, opinonated, but configurable, deployment of the HPCC Systems platform on AKS.

See examples for general usage.


Terraform

Version
>= 1.0.0

Providers

Name Version
azurerm >=2.85.0
helm >=2.1.1
kubernetes >=2.5.0
random >=2.3.0

Inputs

Variable Description Type Default Required
admin_services_storage_account_settings Settings for admin services storage account. object() (see appendix a) {} no
admin_services_storage_size PV sizes for admin service planes (storage billed only as consumed). object() (see appendix b) {} no
data_storage_config HPCC Data storage config. object() (see appendix c) nil yes
enable_node_tuning Enable node tuning daemonset (only needed once per AKS cluster). bool true no
helm_chart_overrides Helm chart values, in yaml format, to be merged last. string nil no
helm_chart_timeout Helm timeout for hpcc chart in seconds. number 600 no
helm_chart_version Version of the HPCC Helm Chart to use. string 8.6.16 no
hpcc_container HPCC container information. object() (see appendix q) nil yes
hpcc_container_registry_auth Registry authentication for HPCC container. object() (see appendix r) nil no
install_blob_csi_driver Install blob-csi-drivers on the cluster. bool true no
ldap_config LDAP settings for dali and esp services. object() (see appendix s) nil no
ldap_tunables Tunable settings for LDAP. string nil no
location Azure region in which to create resources. string nil yes
namespace Kubernetes namespace where resources will be created. object() (see appendix w) hpcc no
node_tuning_containers URIs for containers to be used by node tuning submodule. object() (see appendix x) {} no
node_tuning_container_registry_auth Registry authentication for node tuning containers. object() (see appendix y) {} no
resource_group_name The name of the resource group to deploy resources. string nil yes
roxie_config Settings for roxie service. list(object()) (see appendix z) disabled no
spill_volume_size Storage config for hpcc. string nil no
thor_config Settings for thor service. list(object()) (see appendix cc) disabled no
tags Tags to be applied to Azure resources. map(string) {} no

Appendix A

admin_services_storage_account_settings object specification

Variable Description Type Required
authorized_ip_ranges CIDRs/IPs allowed to access. map(string) yes
delete_protection Enable AzureRM management lock. bool yes
replication_type Storage account Replication. string yes
subnet_ids Service endpoints to create. map(string) yes

Appendix B

admin_services_storage_size object specification

Variable Description Type Required
dali PV/PVC size for dali storage plane. string 100Gi
debug PV/PVC size for debug storage plane. string 100Gi
dll PV/PVC size for dll storage plane. string 100Gi
lz PV/PVC size for lz storage plane. string 1Pi
sasha PV/PVC size for sasha storage plane. string 100Gi

Appendix C

data_storage_config object specification

Variable Description Type Required
internal HPCC data storage provisioned by this module. object() (see appendix D) no
external HPCC data storage provisioned outside this module. object() (see appendix l) yes

Appendix D

data_storage_config.internal object specification

Variable Description Type Required
blob_nfs Blob NFS storage configuration. object() (see appendix e) no
hpc_cache HPC Cache storage configuration. object() (see appendix g) no

Appendix E

data_storage_config.internal.blob_nfs object specification

Variable Description Type Required
data_plane_count Number of data planes (storage accounts/containers) to create. number yes
storage_account_settings Storage account settings for data planes. object() (see appendix f) yes

Appendix F

data_storage_config.internal.blob_nfs.storage_account_settings object specification

Variable Description Type Required
authorized_ip_ranges CIDRs/IPs allowed to access. map(string) yes
delete_protection Enable AzureRM management lock. bool yes
replication_type Storage account Replication. string yes
subnet_ids Service endpoints to create. map(string) yes

Appendix G

data_storage_config.internal.hpc_cache object specification

Variable Description Type Required
dns DNS information. object() (see appendix h) yes
resource_provider_object_id Object ID of HPC Cache resource provider (see appendix i). string yes
size Size of HPC Cache (small, medium, large). string yes
storage_targets Storage target information. map(object()) (see appendix j) yes
subnet_id Virtual network subnet id where HPC Cache will be placed. string yes

Appendix H

data_storage_config.internal.hpc_cache.dns object specification

Variable Description Type Required
zone_name DNS zone name. string yes
zone_resource_group_name Resource group name containting dns zone. string yes

Appendix J

data_storage_config.internal.hpc_cache.resource_provider_object_id sourcing recommendation

This code can be used to retrieve the service principal info:

data "azuread_service_principal" "hpc_cache_resource_provider" {
  display_name = "HPC Cache Resource Provider"
}

The input would then look like this:

resource_provider_object_id = data.azuread_service_principal.hpc_cache_resource_provider.object_id

Appendix J

data_storage_config.internal.hpc_cache.storage_targets object specification

Variable Description Type Required
cache_update_frequency Cache update frequency (never, 30s, 3h). string yes
storage_account_data_planes Storage account data planes. (see appendix k) string yes

Appendix K

data_storage_config.internal.hpc_cache.storage_targets.storage_account_data_planes object specification

Variable Description Type Required
container_id Storage account container id. string yes
container_name Storage account container name. string yes
id Data plane id. number yes
resource_group_name Storage account resource group name. string yes
storage_account_id Storage account id. string yes
storage_account_name Storage account name. string yes

Appendix L

data_storage_config.external object specification

Variable Description Type Required
blob_nfs Blob NFS storage configuration. list(object()) (see appendix m) no
hpc_cache HPC Cache storage configuration. list(object()) (see appendix n) no
hpcc Remote HPCC data configuration. list(object()) (see appendix o) no

Appendix M

data_storage_config.external.blob_nfs object specification

Variable Description Type Required
container_id Storage account container id. string yes
container_name Storage account container name. string yes
id Data plane id. number yes
resource_group_name Storage account resource group name. string yes
storage_account_id Storage account id. string yes
storage_account_name Storage account name. string yes

Appendix N

data_storage_config.external.hpc_cache object specification

Variable Description Type Required
id Data plane id. string yes
path HPC Cache path. string yes
server HPC Cache URI (must be Azure DNS record to ensure full performance). number yes

Appendix O

data_storage_config.external.hpcc object specification

Variable Description Type Required
name Remote HPCC cluster identifier. string yes
planes Data plane information. list(object()) (see appendix p) yes
service Remote HPCC service URI. string yes

Appendix P

data_storage_config.external.hpcc.planes object specification

Variable Description Type Required
local Local data plane name. string yes
remote Remote data plane name. string yes

Appendix Q

hpcc_container object specification

Variable Description Type Required
image_name Name of container image. string yes
image_root URI to image root. string yes
version Container version (null will use helm chart version). string yes

Appendix R

hpcc_container_registry_auth object specification

Variable Description Type Required
password Password/API key. string yes
username Username. string yes

Appendix S

ldap_config object specification

Variable Description Type Required
dali Dali service LDAP settings. object() (see appendix t) yes
esp ESP service LDAP settings. object() (see appendix u) yes
ldap_server LDAP server address. string yes

Appendix T

ldap_config.dali object specification

Variable Description Type Required
adminGroupName LDAP adminGroupName. string yes
filesBasedn LDAP filesBasedn. string yes
groupsBasedn LDAP groupsBasedn. string yes
hpcc_admin_password LDAP hpcc admin password (kubernetes secret will be created). string yes
hpcc_admin_username LDAP hpcc admin username (kubernetes secret will be created). string yes
ldap_admin_password LDAP ldap admin password (kubernetes secret will be created). string yes
ldap_admin_username LDAP ldap admin username (kubernetes secret will be created). string yes
ldapAdminSecretKey LDAP ldapAdminSecretKey. string yes
ldapAdminVaultId LDAP ldapAdminVaultId. string yes
resourcesBasedn LDAP resourcesBasedn. string yes
sudoersBasedn LDAP sudoersBasedn. string yes
systemBasedn LDAP systemBasedn. string yes
usersBasedn LDAP usersBasedn. string yes
workunitsBasedn LDAP workunitsBasedn. string yes

Appendix U

ldap_config.esp object specification

Variable Description Type Required
adminGroupName LDAP adminGroupName. string yes
filesBasedn LDAP filesBasedn. string yes
groupsBasedn LDAP groupsBasedn. string yes
ldap_admin_password LDAP ldap admin password (kubernetes secret will be created). string yes
ldap_admin_username LDAP ldap admin username (kubernetes secret will be created). string yes
ldapAdminSecretKey LDAP ldapAdminSecretKey. string yes
ldapAdminVaultId LDAP ldapAdminVaultId. string yes
resourcesBasedn LDAP resourcesBasedn. string yes
sudoersBasedn LDAP sudoersBasedn. string yes
systemBasedn LDAP systemBasedn. string yes
usersBasedn LDAP usersBasedn. string yes
workunitsBasedn LDAP workunitsBasedn. string yes

Appendix V

ldap_tunables object specification

Variable Description Type Default Required
cacheTimeout LDAP adminGroupName. number 5 yes
checkScopeScans LDAP filesBasedn. bool true yes
ldapTimeoutSecs LDAP groupsBasedn. number 131 yes
maxConnections LDAP hpccAdminSecretKey. number 10 yes
passwordExpirationWarningDays LDAP ldapAdminSecretKey. number 10 yes
sharedCache LDAP ldapAdminVaultId. bool true yes

Appendix W

namespace object specification

Variable Description Type Default Required
namespace Namespace name. string hpcc yes
labels Lables to be applied to the namespace'. map(string) {name = "hpcc"} no

Appendix X

node_tuning_containers object specification

Variable Description Type Default Required
busybox URI for busybox container. string docker.io/library/busybox:1.34 yes
debian URI for debian container (slim preferred)'. string docker.io/library/debian:bullseye-slim yes

Appendix Y

node_tuning_container_registry_auth object specification

Variable Description Type Required
password Password/API key. string yes
username Username. string yes

Appendix Z

roxie_config object specification

Variable Description Type Required
disabled Disable this roxie config. bool yes
name Name of roxie config. string yes
numChannels Number of pods per cluster. number yes
prefix Root directory for access plane. string yes
replicas Number of replicas per channel. number yes
serverReplicas Number of replica sets. number yes
services Service configs. list(object()) (see appendix aa) yes
topoServer TopoServer config. object() (see appendix bb) yes

Appendix AA

roxie_config.services object specification

Variable Description Type Required
name Service name. string yes
servicePort Service port. number yes
listenQueue Listen queue length. number yes
numThreads Number of threads. number yes
visability Service visability. string yes

Appendix BB

roxie_config.topoServer object specification

Variable Description Type Required
replicas Number of replicas. number yes

Appendix CC

thor_config object specification

Variable Description Type Required
disabled Disable this Thor config. bool yes
eclAgentResources ECL Agent resource settings. object() (see appendix dd) yes
keepJobs Persist pods after job failure. string ("none" or "all") yes
managerResources Manager resource settings. object() (see appendix ee) yes
maxGraphs Maximum number of graphs. number yes
maxJobs Maximum number of jobs in queue. number yes
name Name of Thor config. string yes
numWorkersPerPod Number of workers per pod. number yes
numWorkers Number of Thor workers. number yes
prefix Root directory for access plane. string yes
workerMemory Worker memory settings. object() (see appendix ff) yes
workerResources Worker resource settings. object() (see appendix gg) yes

Appendix DD

thor_config.eclAgentResources object specification

Variable Description Type Required
cpu CPU config. string yes
memory Memory config. string yes

Appendix EE

thor_config.managerResources object specification

Variable Description Type Required
cpu CPU config. string yes
memory Memory config. string yes

Appendix FF

thor_config.workerMemory object specification

Variable Description Type Required
query Query memory config. string yes
thirdParty Third party memory config. string yes

Appendix GG

thor_config.workerResources object specification

Variable Description Type Required
cpu CPU config. string yes
memory Memory config. string yes

About

This is a copy of my repository on my vm2 where I've made changed to godji's opinionated-hpcc terraform.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages