Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust all Implementation manifests to use latest workflow syntax and storage backends #666

Closed
Tracked by #604
pkosiec opened this issue Mar 15, 2022 · 2 comments
Closed
Tracked by #604
Assignees
Labels
area/hub-manifests Relates to Hub manifests enhancement New feature or request
Milestone

Comments

@pkosiec
Copy link
Member

pkosiec commented Mar 15, 2022

Description

Blocked by #634, #650, #665

Additional scope:

  • Move outputting static Helm Release logic from Helm Runner inputs to env variables
  • Remove output.useHelmReleaseStorage from helm.install and add env variable to Helm Runner container. We have now two Implementations (install and install-static, so we can distinguish them). However, all manifests that use Helm runner should be updated to always assume they use the new helm.install which outputs dynamic Helm Release TI. That means:
capact-outputTypeInstances:
  - name: helm-release
     from: helm-release

Needs to be changed to:

capact-outputTypeInstances:
  - name: helm-release
     from: helm-release
     backend: helm-release-storage # from requires section
  • Remove output.useHelmReleaseStorage from helm.upgrade input Type - detect input TypeInstance in Helm Runner and output the same shape
  • Consider making a dedicated Interface and Implementation for the ti-value-fetcher.

Timebox for 3SP.

Related issues

See epic #604 for reason and use cases.

@pkosiec pkosiec added enhancement New feature or request area/hub-manifests Relates to Hub manifests labels Mar 15, 2022
@pkosiec pkosiec changed the title Adjust all Implementation manifests to use latest workflow syntax (Argo workflows artifact content) [13MD / timebox 8MD] Adjust all Implementation manifests to use latest workflow syntax and storage backends Mar 15, 2022
@pkosiec pkosiec added this to the 0.7.0 milestone Apr 12, 2022
@mszostok mszostok self-assigned this Apr 13, 2022
@mszostok
Copy link
Member

mszostok commented Apr 21, 2022

Testing scenarios

AWS:

  • Elasticsearch - tested separately ✅
  • Redis - tested separately ✅
  • RDS - tested as a part of Mattermost install ✅
  • Secrets Manager - tested manually ✅

GCP:

  • CloudSQL - tested as a part of Concourse install ✅

Bitnami:

  • MinIO - tested separately ✅
  • PostgreSQL - tested as a part of Mattermost install ✅
  • Redis - tested separately ✅
  • MongoDB - tested as a part of RocketChat install ✅

Elastic:

  • Elasticsearch - tested separately ✅ - as the only one I left it with the static Helm storages

Capact:

  • Upgrade - the Helm storage is not available when the Capact is installed, as a result, we cannot change Upgrade action to dynamic Helm storage.

Applications:

  1. Install Mattermost
    1. RDS - found issue that will be solved by Storage backend ID for TypeInstance from nested workflow is lost during TypeInstance upload #705
    2. on-prem PostgreSQL ✅
  2. Install RocketChat
    1. on-prem MongoDB ✅
  3. Install Concourse
    1. CloudSQL - in progress ✅

Helm Storage

Found Issues:

  • We have memory issue, if we have multiple TypeInstances with Template backend that refers to the bitnami Helm chart. Additionally, executing capact ti get -oyaml causes that the Helm Template backend pod is OOMKilled because in a single request we fetch all TypeInstance, but the same instance is “fetched” multiple times because of uses relation and latest previous revisions revision field. The Helm Template backend probably doesn't have memory leak as the resources are released. Later it has higher memory usage, but this could be due to memory management (because of performance reason sometimes app doesn't release all memory until it is asked explicitly by the system)

    Screen Shot 2022-04-21 at 14 08 37

I installed the Helm storage via Dashboard and applied such Global Policy:

cat > /tmp/helm-storage-policy.yaml << ENDOFFILE
interface:
  default: # properties applied to all rules above
     inject:
       requiredTypeInstances:
          - id: "3e4af86a-ab76-48c5-8799-1bcb0d994018"
            description: "Helm template"
          - id: "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
            description: "Helm release"
  rules:
      - interface:
          path: cap.*
        oneOf:
        - implementationConstraints:
            requires:
            - path: cap.core.type.platform.kubernetes
        - implementationConstraints: {}
ENDOFFILE
capact policy apply -f /tmp/helm-storage-policy.yaml

Elasticsearch

Found issues:

  • The host parameters is ignored in AWS Implementation. However, the output TypeInstance contains the Elastic URL from AWS ELB which is the valid one.
  • The version is different. In AWS Implementation I changed it from number to string, however we are not able to enforce the SemVer format, we got 7.10 but we want 7.10.0.
    I decided to remove the SemVer regex. We should not force it on AWS Implementation.

AWS

  1. Create TypeInstance with AWS Credentials.
  2. Create Action Policy to prefer AWS Implementation:
    export AWS_TI_ID="76db17cc-6a72-4d3a-99d7-4c6cbf887446"
    cat > /tmp/action-policy.yaml << ENDOFFILE
    interface:
      rules:
        - interface:
            path: "cap.interface.analytics.elasticsearch.install"
          oneOf:
            - implementationConstraints:
                attributes:
                  - path: "cap.attribute.cloud.provider.aws"
              inject:
                requiredTypeInstances:
                - id: ${AWS_TI_ID}
                  description: "AWS credentials"
        - interface:
            path: "cap.*"
          oneOf:
            - implementationConstraints:
                requires:
                  - path: "cap.core.type.platform.kubernetes"
            - implementationConstraints: {}
    ENDOFFILE
  3. Create input parameters:
    cat > /tmp/aws-elastic.yaml << ENDOFFILE
    input-parameters:
      host: "aws-elastic.capact.local" # this is ignored by AWS implementation
      replicas: "1"
    ENDOFFILE
  4. Create an Action:
    capact act create cap.interface.analytics.elasticsearch.install --name "aws-elastic" --parameters-from-file /tmp/aws-elastic.yaml --action-policy-from-file /tmp/action-policy.yaml
  5. Run and watch Action:
    capact act run aws-elastic
    capact act watch aws-elastic

Test

  1. Get TypeInstance details:

    capact ti get 0fbb0212-bc2e-47de-a971-dbcbf46b39af -ojson | jq -r '.[0].latestResourceVersion.spec.value'
    

    Expected output:

    {
      "basicAuthSecretName": "elastic-basic-authr8gx5",
      "host": "search-elastic-rntjuhjurdwz-wubimtrl2egzfwzrufcxcqc3bu.eu-west-1.es.amazonaws.com",
      "port": 443,
      "protocol": "https",
      "version": "7.10"
    }
  2. Check health:

    export USER=$(kubectl -n default get secret elastic-basic-authr8gx5  -ogo-template='{{.data.username | base64decode }}')
    export PASS=$(kubectl -n default get secret elastic-basic-authr8gx5  -ogo-template='{{.data.password | base64decode }}')
    
    curl -u $USER:$PASS https://search-elastic-rntjuhjurdwz-wubimtrl2egzfwzrufcxcqc3bu.eu-west-1.es.amazonaws.com/_cat/health

Delete

To delete resource on AWS side, run:

curl -LO https://storage.googleapis.com/capactio-terraform-modules/terraform.aws.elasticsearch.provision/0.1.0/module.tgz
tar zxvf module.tgz

capact typeinstance get 0fbb0212-bc2e-47de-a971-dbcbf46b39af -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
capact typeinstance get 0fbb0212-bc2e-47de-a971-dbcbf46b39af -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars

terraform init
terraform destroy # region eu-west-1

Helm chart

Install it via Dashboard.

Test

  1. Get TypeInstance details:

    capact ti get b02c9803-f7f3-4c7d-b663-9b4c634986e6 -ojson | jq -r '.[0].latestResourceVersion.spec.value'

    Expected output:

    {
    "basicAuthSecretName": "elastic-basic-authmxpvp",
    "host": "elastic.capact.local",
    "port": 80,
    "protocol": "http",
    "version": "7.4.1"
    }
  2. Check health:

    export USER=$(kubectl -n default get secret elastic-basic-authmxpvp  -ogo-template='{{.data.username | base64decode }}')
    export PASS=$(kubectl -n default get secret elastic-basic-authmxpvp  -ogo-template='{{.data.password | base64decode }}')
    
    curl -u $USER:$PASS https://elastic.capact.local/_cat/health

Delete

It's not needed. Resources will be removed on cluster deprovisioning.

Redis

Found issues:

  • high memory consumption on Helm Template storage > 1.5G.
  • In AWS Implementation the vpc_id, subnet_ids and node_type are required on additional parameters. This doesn't seem to be right. Our Action will be rendered but during execution it will fail with such error:
    2022-04-21T12:26:27.707Z	DEBUG	runner.terraform	terraform/terraform.go:103	Terraform output	{"output": "var.node_type\n  The compute and memory capacity of the nodes in the node group.\n\n  Enter a value: var.subnet_ids\n  List of VPC Subnet IDs for the cache subnet group.\n\n  Enter a value: var.vpc_id\n  VPC Id to associate with Redis ElastiCache.\n\n  Enter a value: \nError: No value for required variable\n\n  on variables.tf line 18:\n  18: variable \"node_type\" {\n\nThe root module input variable \"node_type\" is not set, and has no default\nvalue. Use a -var or -var-file command line argument to provide a value for\nthis variable.\n\nError: No value for required variable\n\n  on variables.tf line 23:\n  23: variable \"subnet_ids\" {\n\nThe root module input variable \"subnet_ids\" is not set, and has no default\nvalue. Use a -var or -var-file command line argument to provide a value for\nthis variable.\n\nError: No value for required variable\n\n  on variables.tf line 28:\n  28: variable \"vpc_id\" {\n\nThe root module input variable \"vpc_id\" is not set, and has no default value.\nUse a -var or -var-file command line argument to provide a value for this\nvariable.\n"}
    2022-04-21T12:26:27.708Z	ERROR	runner.terraform	runner/manager.go:77	while waiting for runner completion	{"runner": "terraform", "dryRun": false, "error": "terraform failed to finish: while running terraform: exit status 1", "errorVerbose": "exit status 1\nwhile running terraform\ncapact.io/capact/pkg/runner/terraform.(*terraform).Start.func1\n\t/capact.io/capact/pkg/runner/terraform/terraform.go:53\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nterraform failed to finish\ncapact.io/capact/pkg/runner/terraform.(*terraformRunner).WaitForCompletion\n\t/capact.io/capact/pkg/runner/terraform/runner.go:100\ncapact.io/capact/pkg/runner.(*Manager).Execute\n\t/capact.io/capact/pkg/runner/manager.go:75\nmain.main\n\t/capact.io/capact/cmd/terraform-runner/main.go:29\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}
    capact.io/capact/pkg/runner.(*Manager).Execute
      /capact.io/capact/pkg/runner/manager.go:77
    main.main
      /capact.io/capact/cmd/terraform-runner/main.go:29
    runtime.main
      /usr/local/go/src/runtime/proc.go:225
    2022/04/21 12:26:27 while executing runner: while waiting for completion: terraform failed to finish: while running terraform: exit status 1
    
    However, I didn't fix that. I just provided the defaults for a given region.
  • In AWS Implementation the input password must contain from 16 to 128 alphanumeric characters or symbols (excluding @, ", and /). I added "minLength": 16, property to JSON Schema. We can also prepare the regex for it but this would be more problematic to maintain.

AWS

  1. Create TypeInstance with AWS Credentials.

  2. Create Action Policy to prefer AWS Implementation

    export AWS_TI_ID="76db17cc-6a72-4d3a-99d7-4c6cbf887446"
    cat > /tmp/action-policy.yaml << ENDOFFILE
    interface:
      rules:
        - interface:
            path: "cap.interface.database.redis.install"
          oneOf:
            - implementationConstraints:
                attributes:
                  - path: "cap.attribute.cloud.provider.aws"
              inject:
                requiredTypeInstances:
                - id: ${AWS_TI_ID}
                  description: "AWS credentials"
                additionalParameters:
                  - name: additional-parameters
                    value: # default for region eu-west-1
                      vpc_id: vpc-03c9859997d2e6850
                      subnet_ids:
                        - subnet-0286ffe48cefeb287
                        - subnet-0608e53e751752501
                        - subnet-023e6ddbe6ffa9583
                      node_type: cache.t3.small
        - interface:
            path: "cap.*"
          oneOf:
            - implementationConstraints:
                requires:
                  - path: "cap.core.type.platform.kubernetes"
            - implementationConstraints: {}
    ENDOFFILE
  3. Create input parameters:

    cat > /tmp/aws-redis.yaml << ENDOFFILE
    input-parameters:
      name: "aws-redis-instance"
      password: ...trimmed...
    ENDOFFILE
  4. Create an Action:

    capact act create cap.interface.database.redis.install --name "aws-redis" --parameters-from-file /tmp/aws-redis.yaml --action-policy-from-file /tmp/action-policy.yaml
  5. Run and watch Action:

    capact act run aws-redis
    capact act watch aws-redis

Test

  1. Get Action output:

    capact act get aws-redis -ojson | jq -r '.Actions[0].output.typeInstances'
  2. Get TypeInstance details:

    capact ti get cb17e406-26ad-4c25-8996-1d23879fc8f7 -ojson | jq -r '.[0].latestResourceVersion.spec.value'
    

    Expected output:

    {
      "host": "master.aws-redis-instance-redis.jin6nd.euw1.cache.amazonaws.com",
      "password": "...trimmed...",
      "port": 6379
    }

I tried to reach the Redis instance via redis-cli but without success. However, we can see that it is up and running on AWS console. I also created a dedicated EC2 and check the connection between EC2 and Redis node, see: https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1#NetworkPath:pathId=nip-0e75c9c53af9c2a8f. As you can see the connection is possible. Maybe Redis itself deny the request. I ran out of time for further debugging. We also don't have a case where Redis is used.

Delete

To delete resource on AWS side, run:

git clone --depth 1 https://github.com/umotif-public/terraform-aws-elasticache-redis.git
cd terraform-aws-elasticache-redis

capact typeinstance get 9a1b2b10-539f-4abf-9e05-5cf3e41ac200 -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
capact typeinstance get 9a1b2b10-539f-4abf-9e05-5cf3e41ac200 -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars

terraform init
terraform destroy # region eu-west-1

Next delete the EC2 https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#InstanceDetails:instanceId=i-0c21aa8a04b0f7bde and analysis https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1#NetworkPath:pathId=nip-0e75c9c53af9c2a8f

Helm chart

  1. Install Helm Storage and configure Global Policy
  2. Install Redis via Dashboard. The Helm Implementation will be selected.

Test

  1. Get Action output:
    capact act get local-redis -ojson | jq -r '.Actions[0].output'
    Expected output:
    {
      "typeInstances": [
        {
          "backend": {
            "abstract": false,
            "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
          },
          "id": "bcad8324-5148-4a83-aab3-b2a0da7afd87",
          "typeRef": {
            "path": "cap.type.database.redis.config",
            "revision": "0.1.0"
          }
        },
        {
          "backend": {
            "abstract": false,
            "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
          },
          "id": "5a7ec310-94ab-4b57-a5ee-ad241c128e99",
          "typeRef": {
            "path": "cap.type.helm.chart.release",
            "revision": "0.1.0"
          }
        }
      ]
    }
    As you can see, data is stored in external backends.

Rocket.Chat

Helm chart

  1. Install Helm Storage and configure Global Policy
  2. Install Rocket.Chat via Dashboard. The Helm Implementation will be selected.

Test

  1. Create tunel:

    ssh -f -M -N -S /tmp/rocket.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:8088:rocket.capact.local:443
  2. Update /etc/hosts on your localhost:

    127.0.0.1 rocket.capact.local
  3. Navigate to https://rocket.capact.local:8088/

    Username: admin
    Password: admin
    
  4. Get Action output:

    capact act get local-rocket -ojson | jq -r '.Actions[0].output.typeInstances'

    Expected output:

    [
      {
        "backend": {
          "abstract": false,
          "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
        },
        "id": "74502708-331e-47ea-beaa-9fb2a1d2764b",
        "typeRef": {
          "path": "cap.type.database.mongodb.config",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
        },
        "id": "1448c96d-8391-4752-9bce-2e6d0ee6fbc0",
        "typeRef": {
          "path": "cap.type.helm.chart.release",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
        },
        "id": "83ab4f5c-9190-4bec-baba-3fcdea85d665",
        "typeRef": {
          "path": "cap.type.helm.chart.release",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
        },
        "id": "365ce52c-1bca-463c-b76b-af32cb0b15a6",
        "typeRef": {
          "path": "cap.type.productivity.rocketchat.config",
          "revision": "0.1.0"
        }
      }
    ]

    As you can see, data is stored in external backends.

  5. You can check all of them by running:

    capact ti get "74502708-331e-47ea-beaa-9fb2a1d2764b" "1448c96d-8391-4752-9bce-2e6d0ee6fbc0" "83ab4f5c-9190-4bec-baba-3fcdea85d665" "365ce52c-1bca-463c-b76b-af32cb0b15a6" -ojson | jq -r '.[].latestResourceVersion.spec.value'

    NOTE: This may take up to 1 min.

Delete

It's not needed. Resources will be removed on cluster deprovisioning.

Mattermost

PSQL Helm chart

  1. Install Helm Storage and configure Global Policy
  2. Install Mattermost via Dashboard. The Helm Implementation will be selected.

Test

  1. Create tunel:

    ssh -f -M -N -S /tmp/local-mattermost.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:8089:mattermost.capact.local:443
  2. Update /etc/hosts on localhost:

    127.0.0.1 mattermost.capact.local
  3. Navigate to: https://mattermost.capact.local:8089/

    Username: admin
    Password: admin
    
  4. Get Action output:

    capact act get local-mattermost -ojson | jq -r '.Actions[0].output.typeInstances'

    Expected output:

    [
      {
        "backend": {
          "abstract": false,
          "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
        },
        "id": "e71d0e84-051d-4c99-a682-30673d5167a5",
        "typeRef": {
          "path": "cap.type.database.postgresql.config",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "642a5244-0be6-40bb-99f3-06c71e23d2f8",
        "typeRef": {
          "path": "cap.type.database.postgresql.database",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "2c5c7fe1-589a-4343-a534-92923c771c7f",
        "typeRef": {
          "path": "cap.type.database.postgresql.user",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
        },
        "id": "ca43ae2d-f112-4719-90ae-3b186160e45f",
        "typeRef": {
          "path": "cap.type.helm.chart.release",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
        },
        "id": "db12a941-6e0c-4c12-88b5-90e8cd26ec93",
        "typeRef": {
          "path": "cap.type.helm.chart.release",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
        },
        "id": "ba5379b0-2cee-411a-9ad8-03364520e53c",
        "typeRef": {
          "path": "cap.type.productivity.mattermost.config",
          "revision": "0.1.0"
        }
      }
    ]

    As you can see, data is stored in external backends.

  5. You can check Mattermost config by running:

    capact ti get -ojson ba5379b0-2cee-411a-9ad8-03364520e53c | jq -r '.[0].latestResourceVersion.spec.value'

    NOTE: This may take up to 30sec.

  6. Clean-up:

    ssh -S /tmp/local-mattermost.vm.sock -O exit 18.184.179.238

Delete

It's not needed. Resources will be removed on cluster deprovisioning.

RDS

Found issues:

  • We have such property port: {{ .port }} in goTemplate property. Unfortunately this generates such problem:
    2022/04/21 14:26:44 while executing runner: while reading runner input: while unmarshaling data from file "/terraform-args": error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".port":interface {}(nil)}
    However, we cannot quote it as in the output we expect the number and not string. To fix this problem, I had to address this issue: Change output.goTemplate in Terraform-based Implementations to be of type string #504.
  • Another problem was the PostreSQL version:
    Warning: Argument is deprecated
    
      with module.db.module.db_instance.aws_db_instance.this,
      on .terraform/modules/db/modules/db_instance/main.tf line 34, in resource \"aws_db_instance\" \"this\":
      34:   name                                = var.name
    
    Use db_name instead
    
    (and 3 more similar warnings elsewhere)
    
    Error: Error creating DB Instance: InvalidParameterCombination: Cannot find version 11.10 for postgres
    \tstatus code: 400, request id: 0bb0f5eb-740f-4cf4-b5d6-7a7475cb7036
    
      with module.db.module.db_instance.aws_db_instance.this[0],
      on .terraform/modules/db/modules/db_instance/main.tf line 20, in resource \"aws_db_instance\" \"this\":
      20: resource \"aws_db_instance\" \"this\" {
    
    "}
    Was fixed by changing the default version directly in Implementation manifest.
  • Two Mattermost installation cannot be in the same Namespace. There is a name conflict with created SA.
  • For now we are not able to change the PostgreSQL Implemenation to RDS due to Storage backend ID for TypeInstance from nested workflow is lost during TypeInstance upload #705
                  # Install DB
                  - - name: install-db
                      capact-when: postgresql == nil
                      capact-action: postgresql.install
                      capact-outputTypeInstances:
                        - name: postgresql
                          from: postgresql
                          # Affected by https://github.com/capactio/capact/issues/705
                          # This needs to be removed to allow other non-Helm based Implementation
                          backend: helm-template-storage
    However, the installation works propertly. It's only a matter of outputting the TypeInstances.

Follow Install Mattermost with an external AWS RDS database tutorial. In my case it was:

  1. Policy:
    export AWS_TI_ID="76db17cc-6a72-4d3a-99d7-4c6cbf887446"
    cat > /tmp/policy.yaml << ENDOFFILE
    interface:
      default:
        inject:
          requiredTypeInstances:
            - id: "3e4af86a-ab76-48c5-8799-1bcb0d994018"
              description: "Helm template"
            - id: "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
              description: "Helm release"
      rules:
        - interface:
            path: cap.interface.database.postgresql.install
          oneOf:
             - implementationConstraints:
                 attributes:
                   - path: "cap.attribute.cloud.provider.aws"
               inject:
                 requiredTypeInstances:
                 - id: ${AWS_TI_ID}
                   description: "AWS credentials"
        - interface:
            path: cap.*
          oneOf:
            - implementationConstraints:
                requires:
                  - path: "cap.core.type.platform.kubernetes"
            - implementationConstraints: {} # fallback to any Implementation
    ENDOFFILE
  2. Input params:
    cat <<EOF > /tmp/mattermost-install.yaml
    input-parameters:
      host: rds-mattermost.capact.local
    EOF
  3. Action:
    kubectl create ns  rds-mattermost
    capact action create --namespace rds-mattermost --name rds-mattermost cap.interface.productivity.mattermost.install --parameters-from-file /tmp/mattermost-install.yaml --action-policy-from-file /tmp/policy.yaml

Delete

To delete resource on AWS side, run:

curl -LO https://storage.googleapis.com/capactio-terraform-modules/terraform.aws.rds.postgresql.provision/0.1.0/module.tgz
tar zxvf module.tgz

capact typeinstance get 681bd80b-1079-4c7d-b0ae-45fffb93c548 -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
capact typeinstance get 681bd80b-1079-4c7d-b0ae-45fffb93c548 -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars

terraform init
terraform destroy # region eu-west-1

Concourse

Found issues:

  • Missing unpackValue: true for rendering GCP project name.
  • Wrong input credentials syntax
nError: Attempted to load application default credentials since neither `credentials` nor `access_token` was set in the provider block.  No credentials loaded. To use your gcloud credentials, run 'gcloud auth application-default login'.  Original error: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: missing 'type' field in credentials

  with provider[\"registry.terraform.io/hashicorp/google\"],
    on main.tf line 46, in provider \"google\":
      46: provider \"google\" {

        "
      }
  • Argo race issue:
         └───✖ helm-install                                                               concourse-install-helm-install-helm                                                                                    child 'concourse-3698900586' failed
             ├───✔ inject-input-type-instances-e0489c2c-8005-42f4-80ae-538c0c7d7cc2-step  inject-input-type-instances-e0489c2c-8005-42f4-80ae-538c0c7d7cc2                       concourse-2371691802  2s
             ├───✔ helm-install                                                           concourse-install-helm-install-helm-install                                            concourse-19934320    2m
             └───⚠ resolve-helm-rel-value                                                 concourse-install-helm-install-resolve-ti-art-value                                    concourse-3698900586  6s        Error (exit code 1): cannot enter chroot for container named "main": no PID known - maybe short running container
    To solve that issue, I modified the value-fetcher to run for at least 1 sec.

Cloud SQL

  1. Create TypeInstance with GCP Credentials.

  2. Action Policy:

    cat > /tmp/cloudsqlpolicy.yaml << ENDOFFILE
    interface:
      default:
        inject:
          requiredTypeInstances:
            - id: "3e4af86a-ab76-48c5-8799-1bcb0d994018"
              description: "Helm template"
            - id: "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
              description: "Helm release"
      rules:
        - interface:
            path: cap.interface.database.postgresql.install
          oneOf:
            - implementationConstraints:
                attributes:
                  - path: "cap.attribute.cloud.provider.gcp"
                requires:
                  - path: "cap.type.gcp.auth.service-account"
              inject:
                requiredTypeInstances:
                  - id: 29f9a769-efa6-4880-854f-187a6b682e18
                    description: "GCP Service Account"
        - interface:
            path: cap.*
          oneOf:
            - implementationConstraints:
                requires:
                  - path: "cap.core.type.platform.kubernetes"
            - implementationConstraints: {} # fallback to any Implementation
    ENDOFFILE
  3. Input params:

    cat <<EOF > /tmp/concourse-install.yaml
    input-parameters:
      host: concourse.capact.local
    EOF
  4. Action:

    kubectl create ns  cloud-concourse
    capact action create --namespace cloud-concourse --name concourse cap.interface.automation.concourse.install --parameters-from-file /tmp/concourse-install.yaml --action-policy-from-file /tmp/cloudsqlpolicy.yaml

Test

  1. Create tunel:

    ssh -f -M -N -S /tmp/cloud-concourse.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:8022:concourse.capact.local:443
  2. Update /etc/hosts on localhost:

     127.0.0.1 concourse.capact.local
  3. Navigate to https://concourse.capact.local:8022/.

  4. Get Action output:

    capact act get concourse -n cloud-concourse -ojson | jq -r '.Actions[0].output.typeInstances'

    Expected output:

    [
       {
         "backend": {
           "abstract": false,
           "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
         },
         "id": "84af3e01-0c6a-4046-adde-bf7b9ac79e05",
         "typeRef": {
           "path": "cap.type.automation.concourse.config",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": true,
           "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
         },
         "id": "e55db908-a1d2-4e6c-81d4-e8676d6d1bf5",
         "typeRef": {
           "path": "cap.type.database.postgresql.config",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": true,
           "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
         },
         "id": "d8bcdb6b-f857-4c38-bffe-b20dae9de914",
         "typeRef": {
           "path": "cap.type.database.postgresql.database",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": true,
           "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
         },
         "id": "eca6ca74-75d0-405a-a4a8-ad5be04fa91b",
         "typeRef": {
           "path": "cap.type.database.postgresql.user",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": false,
           "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
         },
         "id": "7a8132b1-8c41-492b-8b4e-178640926882",
         "typeRef": {
           "path": "cap.type.helm.chart.release",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": true,
           "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
         },
         "id": "fec082e7-6b03-4f32-865f-922809959d18",
         "typeRef": {
           "path": "cap.type.terraform.release",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": true,
           "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
         },
         "id": "e486b409-622d-4978-bc12-0eba8ea3ea05",
         "typeRef": {
           "path": "cap.type.terraform.state",
           "revision": "0.1.0"
         }
       }
     ]

    As you can see, data is stored in external backends.

  5. You can check Concourse config by running:

     capact ti get -ojson 84af3e01-0c6a-4046-adde-bf7b9ac79e05 | jq -r '.[0].latestResourceVersion.spec.value'

    NOTE: This may take up to 30sec.

  6. Clean-up:

     ssh -S /tmp/cloud-concourse.vm.sock -O exit 18.184.179.238

Delete

To delete resource on GCP side:

  1. Delete the Cloud SQL manually from GCP dashboard, navigate to https://console.cloud.google.com/sql/instances?project=capact.

  2. Ensure that all resources are deleted, run:

    curl -LO https://storage.googleapis.com/capactio-terraform-modules/terraform.gcp.cloudsql.postgresql.install/0.2.0/module.tgz
    tar zxvf module.tgz
    
    capact typeinstance get abe156ca-dfd0-4c05-b213-7bd880865dce -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
    capact typeinstance get abe156ca-dfd0-4c05-b213-7bd880865dce -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars
    
    terraform init
    terraform destroy # region eu-west-1

NOTE Unfortunately running the TF is not possible as the deletion_protection is enabled. You can run TF to change that setting but it's easier to do that via GCP dashboard.

MinIO

Issues:

  • Argo time race:
             └───⚠ resolve-helm-rel-value                                                 deploy-helm-install-resolve-ti-art-value                          local-minio-3788856621  6s        Error (exit code 1): cannot enter chroot for container named "main": no PID known - maybe short running container
    This happens when the value is already provided:
    $ kc logs local-minio-3788856621 main
    2022-04-27T09:42:53.763Z	INFO	ti-value-fetcher	ti-value-fetcher/fetcher.go:48	Value already provided. Skipping...
    To solve that issue, I modified the value-fetcher to run for at least 1 sec.

Helm chart

  1. Install Helm Storage and configure Global Policy
  2. Install MinIO via Dashboard. The Helm Implementation will be selected.

Test

  1. Get Action output:
    capact act get local-minio -ojson | jq -r '.Actions[0].output.typeInstances'
    Expected output:
    [
      {
        "backend": {
          "abstract": false,
          "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
        },
        "id": "6105fad9-94e0-465c-a690-6e979e3118cb",
        "typeRef": {
          "path": "cap.type.database.object.aws.s3.config",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
        },
        "id": "73716174-7fc0-4e7f-8236-1b1965133ea6",
        "typeRef": {
          "path": "cap.type.helm.chart.release",
          "revision": "0.1.0"
        }
      }
    ]
    As you can see, data is stored in external backends.
  2. You can check MinIO config by running:
     capact ti get -ojson 6105fad9-94e0-465c-a690-6e979e3118cb | jq -r '.[0].latestResourceVersion.spec.value'

    NOTE: This may take up to 30sec.

  3. Run MinIO client Pod:
    kubectl run my-mc --rm -i --tty --image minio/mc:latest --command -- bash
  4. Login:
    mc alias set myminio/ http://local-s3-bucket.default.svc.cluster.local:9000 hsltgwucrnfsnwsi zwmbsifbrtdzohgj
  5. Create, list and delete bucket:
    mc mb myminio/test
    mc ls myminio/
    mc rb myminio/test

Appendix

Forward Gateway

Create local tunel to Gateway.

  1. Run on localhost:

    ssh -f -M -N -S /tmp/gateway.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:8082:gateway.capact.local:443
  2. Log in into Gateway:

    capact login https://gateway.capact.local:8082 -u graphql -p t0p_s3cr3t

Forward Dashboard

Create local tunel to Dashboard.

  1. Run on localhost:

    ssh -f -M -N -S /tmp/dashboard.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:8081:dashboard.capact.local:443
  2. Navigate to https://dashboard.capact.local:8081.

Forward Neo4j browser

  1. Run on VM:

    kubectl -n capact-system port-forward svc/neo4j-neo4j 7687 7474
  2. Run on localhost:

    ssh -f -M -N -S /tmp/neo1.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:7687:127.0.0.1:7687
    ssh -f -M -N -S /tmp/neo2.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:7474:127.0.0.1:7474
  3. Clean-up:

    ssh -S /tmp/neo1.vm.sock -O exit 18.184.179.238
    ssh -S /tmp/neo2.vm.sock -O exit 18.184.179.238

Forward Grafana

  1. Run on VM:

    export CAPACT_NAMESPACE="capact-system"
    export USERNAME=$(kubectl -n "${CAPACT_NAMESPACE}" get secrets monitoring-grafana -ojsonpath="{.data.admin-user}" | base64 -d)
    export PASSWORD=$(kubectl -n "${CAPACT_NAMESPACE}" get secrets monitoring-grafana -ojsonpath="{.data.admin-password}" | base64 -d)
    
    echo "URL: http://localhost:3000"
    echo "Username: $USERNAME"
    echo "Password: $PASSWORD"
    
    kubectl -n "${CAPACT_NAMESPACE}" port-forward svc/monitoring-grafana 3000:80
  2. Run on localhost:

    ssh -f -M -N -S /tmp/grafana.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:3000:127.0.0.1:3000
  3. Clean-up:

    ssh -S /tmp/grafana.vm.sock -O exit 18.184.179.238

Force manifest repulling

kc exec -it -n capact-system capact-hub-public-5c45558575-7vhlj -c hub-public-populator -- pkill sleep
kc logs -n capact-system capact-hub-public-5c45558575-7vhlj -c hub-public-populator -f

Forward Argo

  1. Run on VM:

    kubectl port-forward -n capact-system svc/argo-argo-workflows-server 2746
  2. Run on localhost:

    ssh -f -M -N -S /tmp/argo.vm.sock -i ~/.ssh/provision-vm-bigbang.pem [email protected] -L 127.0.0.1:2746:127.0.0.1:2746
  3. Clean-up:

    ssh -S /tmp/argo.vm.sock -O exit 18.184.179.238

@mszostok mszostok closed this as completed May 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/hub-manifests Relates to Hub manifests enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants