Nothing happened after finishing the script

[A-Harby]

I followed all the steps in https://manual.grid.tf/documentation/developers/grid_deployment/grid_deployment_full_vm.html#access-the-grid-services to deploy my own stack, created a domain and VM, installed the prerequisites, and filled in the environment file. Then I ran the script, and after finishing downloading and installing, it just did nothing.

This is the script logs.
logs.txt

What should I do now?

[A-Harby]

root@vmv1nju:~/grid_deployment/docker-compose/devnet# docker compose --env-file .secrets.env --env-file .env config
WARN[0000] /root/grid_deployment/docker-compose/devnet/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
name: devnet
services:
  caddy:
    container_name: caddy
    depends_on:
      grid_dashboard:
        condition: service_started
        required: true
    environment:
      DOMAIN: domain.dev.harby.gridtesting.xyz
    image: caddy:2.8.4
    networks:
      tfgrid_bknd: null
    ports:
      - mode: ingress
        target: 80
        published: "80"
        protocol: tcp
      - mode: ingress
        target: 443
        published: "443"
        protocol: tcp
    restart: unless-stopped
    volumes:
      - type: bind
        source: /root/grid_deployment/docker-compose/devnet/Caddyfile
        target: /etc/caddy/Caddyfile
        bind:
          create_host_path: true
      - type: bind
        source: /srv/caddy/data
        target: /data
        bind:
          create_host_path: true
      - type: bind
        source: /srv/caddy/config
        target: /config
        bind:
          create_host_path: true
      - type: bind
        source: /srv/caddy/log
        target: /var/log/caddy
        bind:
          create_host_path: true
  grid_activation_service:
    container_name: grid_activation_service
    depends_on:
      grid_relay:
        condition: service_started
        required: true
    environment:
      ACTIVATION_AMOUNT: "1000000"
      KYC_PUBLIC_KEY: somekey
      MNEMONIC: ***
      URL: ws://tfchain-public-node:9944
    image: ghcr.io/threefoldtech/tfchain_activation_service:2.7.0-rc4
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  grid_dashboard:
    container_name: grid_dashboard
    depends_on:
      grid_proxy:
        condition: service_started
        required: true
    environment:
      ACTIVATION_SERVICE_URL: https://activation.domain.dev.harby.gridtesting.xyz/activation/activate
      BRIDGE_TFT_ADDRESS: GDHJP6TF3UXYXTNEZ2P36J5FH7W4BJJQ4AYYAXC66I2Q2AH5B6O6BCFG
      GRAPHQL_URL: https://graphql.domain.dev.harby.gridtesting.xyz/graphql
      GRIDPROXY_URL: https://gridproxy.domain.dev.harby.gridtesting.xyz
      MANUAL_URL: https://manual.grid.tf/
      MODE: dev
      NETWORK: custom
      PLAYGROUND_URL: https://play.domain.dev.harby.gridtesting.xyz
      RELAY_DOMAIN: wss://relay.domain.dev.harby.gridtesting.xyz
      STATS_URL: https://stats.domain.dev.harby.gridtesting.xyz
      STELLAR_NETWORK: test
      SUBSTRATE_URL: wss://tfchain.domain.dev.harby.gridtesting.xyz
      VERSION: 2.5.2
    image: ghcr.io/threefoldtech/playground:2.5.2
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  grid_proxy:
    container_name: grid_proxy
    depends_on:
      grid_activation_service:
        condition: service_started
        required: true
    environment:
      MNEMONICS: search vague feature catch correct drama involve hello swallow lend orchard resemble
      POSTGRES_DB: tfgrid-graphql
      POSTGRES_HOST: processor_db
      POSTGRES_PASSWORD: postgres
      POSTGRES_PORT: "5432"
      POSTGRES_USER: postgres
      RELAYURL: wss://relay.domain.dev.harby.gridtesting.xyz
      SERVER_PORT: :443
      TFCHAINURL: wss://tfchain.domain.dev.harby.gridtesting.xyz
    image: ghcr.io/threefoldtech/tfgridproxy:0.15.15-dev
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  grid_relay:
    command:
      - --domain
      - relay.domain.dev.harby.gridtesting.xyz
      - -r
      - redis://redis-rmb:6379
      - -s
      - ws://tfchain-public-node:9944
      - -u
      - "1000"
      - -w
      - "500"
      - -l
      - 0.0.0.0:80
      - --debug
    container_name: grid_relay
    depends_on:
      redis-rmb:
        condition: service_started
        required: true
    environment:
      RMB_DOMAIN: relay.domain.dev.harby.gridtesting.xyz
      SUBSTRATE_URL: ws://tfchain-public-node:9944
      USER_PER_WORKER: "1000"
      WORKERS: "500"
    image: ghcr.io/threefoldtech/rmb-relay:v1.3.1
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  grid_stats:
    container_name: grid_stats
    depends_on:
      grid_proxy:
        condition: service_started
        required: true
    image: ghcr.io/threefoldtech/stats:2.5.2
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  indexer_db:
    command:
      - start-single-node
      - --insecure
      - --cache=.45
      - --max-sql-memory=.45
    container_name: indexer_db
    depends_on:
      tfchain-public-node:
        condition: service_started
        required: true
    image: cockroachdb/cockroach:v22.2.15
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
    ulimits:
      nofile:
        soft: 82920
        hard: 82920
    volumes:
      - type: bind
        source: /srv/indexer/
        target: /cockroach/cockroach-data
        bind:
          create_host_path: true
  indexer_db-init:
    command:
      - /setup_db.sh
    container_name: indexer_db-init
    depends_on:
      indexer_db:
        condition: service_started
        required: true
    entrypoint:
      - /bin/bash
    image: cockroachdb/cockroach:v22.2.15
    networks:
      tfgrid_bknd: null
    restart: on-failure
    volumes:
      - type: bind
        source: /root/grid_deployment/docker-compose/devnet/setup_db.sh
        target: /setup_db.sh
        bind:
          create_host_path: true
  indexer_explorer:
    container_name: indexer_explorer
    depends_on:
      indexer_db-init:
        condition: service_completed_successfully
        required: true
    environment:
      DB_HOST: indexer_db
      DB_NAME: defaultdb
      DB_PORT: "26257"
      DB_TYPE: cockroach
      DB_USER: root
    image: subsquid/substrate-explorer:firesquid
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  indexer_gateway:
    command:
      - --database-url
      - postgres://root@indexer_db:26257/defaultdb
      - --scan-max-value
      - "50000"
    container_name: indexer_gateway
    depends_on:
      indexer_db-init:
        condition: service_completed_successfully
        required: true
    environment:
      DATABASE_MAX_CONNECTIONS: "5"
      RUST_LOG: actix_web=info,actix_server=info
    image: subsquid/substrate-gateway:2.7.0
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  indexer_ingest:
    command:
      - -e
      - ws://tfchain-public-node:9944
      - -c
      - "20"
      - --out
      - postgres://root@indexer_db:26257/defaultdb
      - --types-bundle
      - /configs/typesBundle.json
    container_name: indexer_ingest
    depends_on:
      indexer_db-init:
        condition: service_completed_successfully
        required: true
    image: subsquid/substrate-ingest:1
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
    volumes:
      - type: bind
        source: /root/grid_deployment/docker-compose/devnet/typesBundle.json
        target: /configs/typesBundle.json
        bind:
          create_host_path: true
  processor:
    command:
      - yarn
      - process
    container_name: processor
    depends_on:
      processor_db:
        condition: service_started
        required: true
    environment:
      ACTIV_SERV_IMG: 2.7.0-rc4
      CADDY_IMG: 2.8.4
      DASHBOARD_IMG: 2.5.2
      DB_HOST: processor_db
      DB_NAME: tfgrid-graphql
      DB_PASS: postgres
      DB_PORT: "5432"
      DB_USER: postgres
      INDEXER_DB_IMG: v22.2.15
      INDEXER_ENDPOINT_URL: http://indexer_gateway:8000/graphql
      INDEXER_EXPL_IMG: firesquid
      INDEXER_GW_IMG: 2.7.0
      INDEXER_INGEST_IMG: "1"
      POLL_INTERVAL_MS: "300"
      PROC_DB_IMG: "16"
      PROC_IMG: 2.12.2
      PROC_QUERY_IMG: 2.12.2
      PROXY_IMG: 0.15.15-dev
      RMB_IMG: v1.3.1
      RMB_REDIS_IMG: 7.2.3
      SQD_DEBUG: sqd:processor:*
      START_HEIGHT: "0"
      STATS_IMG: 2.5.2
      TFCHAIN_IMG: 2.9.0
      TYPEORM_LOGGING: error
      WS_ENDPOINT: ws://tfchain-public-node:9944
      WS_URL: ws://tfchain-public-node:9944
    image: ghcr.io/threefoldtech/tfchain_graphql_processor:2.12.2
    networks:
      tfgrid_bknd: null
    ports:
      - mode: ingress
        target: 44233
        published: "44233"
        protocol: tcp
    restart: unless-stopped
  processor_db:
    container_name: processor_db
    depends_on:
      indexer_explorer:
        condition: service_started
        required: true
    environment:
      POSTGRES_DB: tfgrid-graphql
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    image: postgres:16
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
    volumes:
      - type: bind
        source: /srv/processor/
        target: /var/lib/postgresql/data
        bind:
          create_host_path: true
  processor_query_node:
    container_name: processor_query_node
    depends_on:
      processor:
        condition: service_started
        required: true
    environment:
      ACTIV_SERV_IMG: 2.7.0-rc4
      CADDY_IMG: 2.8.4
      DASHBOARD_IMG: 2.5.2
      DB_HOST: processor_db
      DB_NAME: tfgrid-graphql
      DB_PASS: postgres
      DB_PORT: "5432"
      DB_USER: postgres
      INDEXER_DB_IMG: v22.2.15
      INDEXER_ENDPOINT_URL: http://indexer_gateway:8000/graphql
      INDEXER_EXPL_IMG: firesquid
      INDEXER_GW_IMG: 2.7.0
      INDEXER_INGEST_IMG: "1"
      PROC_DB_IMG: "16"
      PROC_IMG: 2.12.2
      PROC_QUERY_IMG: 2.12.2
      PROXY_IMG: 0.15.15-dev
      RMB_IMG: v1.3.1
      RMB_REDIS_IMG: 7.2.3
      START_HEIGHT: "0"
      STATS_IMG: 2.5.2
      TFCHAIN_IMG: 2.9.0
      TYPEORM_LOGGING: error
      WS_ENDPOINT: ws://tfchain-public-node:9944
      WS_URL: ws://tfchain-public-node:9944
    image: ghcr.io/threefoldtech/tfchain_graphql_query-node:2.12.2
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  redis-rmb:
    container_name: redis-rmb
    depends_on:
      processor_query_node:
        condition: service_started
        required: true
    image: redis:7.2.3
    networks:
      tfgrid_bknd: null
    restart: unless-stopped
  tfchain-public-node:
    command:
      - --node-key
      - 46069718239286e0ad9cd523a11de64e041c2949ae223ceaf80426331901bf49
      - --base-path
      - /storage
      - --chain
      - /etc/chainspecs/dev/chainSpecRaw.json
      - --port
      - "30333"
      - --bootnodes
      - /ip4/185.206.122.7/tcp/30333/ws/p2p/12D3KooWRdfuKqX8hULMZz521gdqZB2TXJjfrJE5FV71WiuAUrpk
      - --rpc-cors
      - all
      - --prometheus-external
      - --rpc-max-connections=148576
      - --pruning
      - archive
      - --telemetry-url
      - wss://shard1.telemetry.tfchain.grid.tf/submit 1
      - --rpc-methods=Safe
      - --rpc-external
    container_name: tfchain-public-node
    image: ghcr.io/threefoldtech/tfchain:2.9.0
    networks:
      tfgrid_bknd: null
    ports:
      - mode: ingress
        target: 30333
        published: "30333"
        protocol: tcp
    restart: unless-stopped
    volumes:
      - type: bind
        source: /srv/tfchain/
        target: /storage
        bind:
          create_host_path: true
networks:
  tfgrid_bknd:
    name: tfgrid_bknd
    ipam:
      driver: default

[A-Harby]

And I also tried the manual deploy command and got this

root@vmv1nju:~/grid_deployment/docker-compose/devnet# docker compose --env-file .secrets.env --env-file .env up -d
WARN[0000] /root/grid_deployment/docker-compose/devnet/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 77/16
 ⠏ tfchain-public-node [⣿⣿⣿⣿⣿] 59.69MB / 61.1MB  Pulling                                                                                                           22.0s 
 ⠏ grid_activation_service [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 364.7MB / 371.7MB Pulling                                                                                                22.0s 
 ⠏ indexer_explorer [⠀] Pulling                                                                                                                                    22.0s 
 ✔ grid_proxy Pulled                                                                                                                                               10.1s 
 ⠏ grid_dashboard [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 13.91MB / 15.73MB Pulling                                                                                                           22.0s 
 ⠏ indexer_db Pulling                                                                                                                                              22.0s 
 ✔ grid_relay Pulled                                                                                                                                                3.3s 
 ⠏ processor Pulling                                                                                                                                               22.0s 
 ⠏ caddy [⣿⣿⣿⣿] 18.63MB / 18.63MB Pulling                                                                                                                          22.0s 
 ⠏ indexer_gateway [⠀⠀⠀] Pulling                                                                                                                                   22.0s 
 ⠏ indexer_db-init [⣿⣿⣿⣄⣿⣿] 83.43MB / 140MB   Pulling                                                                                                              22.0s 
 ⠏ processor_query_node [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]  91.3MB / 92.91MB Pulling                                                                                                  22.0s 
 ⠏ redis-rmb [⣶⣿⠀⠀⠀⠀⠀] Pulling                                                                                                                                     22.0s 
 ⠏ indexer_ingest [⠀⠀⠀⠀⠀⠀] Pulling                                                                                                                                 22.0s 
 ⠏ processor_db [⣿⣿⣶⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 143.9MB / 153.4MB Pulling                                                                                                         22.0s 
 ⠏ grid_stats [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 16.59MB / 21.46MB Pulling                                                                                                           22.0s 
failed to register layer: lsetxattr security.capability /usr/bin/caddy: operation not supported

[PeterNashaat]

• Issue related to Official Ubuntu 24.04 can not pull certain images zos#2420 not to the installation script itself.
• instead of using ubuntu 24 try using ubuntu 22.

[PeterNashaat]

• @A-Harby There is another work around if you installed docker-io, instead of installing docker-ce .
• Then rerun the installation script again it will work

[PeterNashaat]

@A-Harby it's not safe to share your wallets private keys on a public repo even if it was for a testing env.

[A-Harby]

• @A-Harby There is another work around if you installed docker-io, instead of installing docker-ce .

• Then rerun the installation script again it will work

Okay will try with docker-io.

[A-Harby]

First, I tried the same steps and script with Ubuntu 22 and still got the same error.

Then tried the docker io but also couldn't run it.

Is there something wrong I did?

[coesensbert]

Is there something wrong I did?

No, not regarding this error. It is related to running docker on ubuntu on grid vm's. None grid vm's don't have this issue. It's not related to running the grid backend directly.
See: threefoldtech/zos#2420 (comment)

[A-Harby]

After the help of @PeterNashaat, I was able to deploy the stack successfully.

But unfortunately on IPV6 not on IPV4, so I wasn't able to do any further testing (IPV6 is blocked here) and should be able to continue once https://github.com/threefoldtech/tf_operations/issues/2760 is done.

But found 2 issues:

• The script is not working correctly; it downloads and installs everything, then gets stuck.
  
• Manually installing isn't clear enough, and I still get another error.
  

[A-Harby]

• Proxy, TFchain and RMB are not working after deploying #74

• Playground is outdated and shouldn't be deployed #75

[PeterNashaat]

• @A-Harby for the second issue "Manually installing" what are the steps you took so i can reproduce this issue ?.

[PeterNashaat]

• The first issue is not stuck issue, it's this command tar -I pigz -xf tfchain-mainnet-latest.tar.gz -C /srv/tfchain/chains/tfchain_mainnet/db/, which is decompressing a 222G tfchain-mainnet snapshot which will take some time to finish.

• The second issue here as you were trying to start the env manually you didn't/forgot to create the needed config files which is Caddyfile for the caddy contianer, so after doing command cp Caddyfile-example Caddyfile, containers starts with no errors.

After the help of @PeterNashaat, I was able to deploy the stack successfully.

But unfortunately on IPV6 not on IPV4, so I wasn't able to do any further testing (IPV6 is blocked here) and should be able to continue once threefoldtech/tf_operations#2760 is done.

But found 2 issues:

• The script is not working correctly; it downloads and installs everything, then gets stuck.
  
• Manually installing isn't clear enough, and I still get another error.
  

[PeterNashaat]

• Tried to reinstall tfchain-public node with a new snapshot download and found out that the disk size in not the same as the required space needed for installation

• VM disk has to be 1000GB, and vm is installed with 436GB disk only which is not enough.

• @A-Harby Please recreate the vm with proper specs and run the script again and wait for it to finish.

[A-Harby]

Ok, done https://github.com/threefoldtech/tf_operations/issues/2767 Bert added 2 500 GB disks, but I had to find a way to merge the 3 physical disks into one big disk, 1.5 TB, and I did it using LVM.