Trouble Installing Packs in StackStorm

[ghost]

Hello everyone,

I’m currently running StackStorm on microk8s and have been following the setup instructions from the official documentation. However, I’ve been encountering an issue when trying to install packs, specifically the ‘rabbitmq’ pack.

The command I’ve been using to install the pack is kubectl exec -it stackstorm-ha-1715676993-st2client-7595ccbd6-hnk89 --namespace default -- st2 pack install rabbitmq. However, during the installation process, I’ve been receiving errors. These errors seem to occur at different stages - sometimes at the ‘install_pack_requirements’ stage, and other times at the ‘register_pack’ stage.

I’ve checked the directory /opt/stackstorm/packs and it exists and is writable. Despite this, the pack installations are failing. This issue isn’t limited to the ‘rabbitmq’ pack - I’ve tried with other packs and encountered the same problem.

I’ve also attempted to install the packs directly through the web interface, but unfortunately, I’ve encountered similar errors there as well.

I would appreciate any guidance or suggestions on what might be causing this issue and how to resolve it.
1 case with install_pack_requirements:

st2 pack install rabbitmq

For the "rabbitmq" pack, the following content will be registered:

actions   |  3
rules     |  0
sensors   |  1
aliases   |  0
triggers  |  0

Installation may take a while for packs with many items.

	[ succeeded ] init_task
	[ succeeded ] download_pack
	[ succeeded ] make_a_prerun
	[ succeeded ] get_pack_dependencies
	[ succeeded ] check_dependency_and_conflict_list
	[  failed   ] install_pack_requirements

id: 664335cbc55d80a70cc4428e
action.ref: packs.install
parameters: 
  packs:
  - rabbitmq
status: failed
start_timestamp: Tue, 14 May 2024 09:58:35 UTC
end_timestamp: Tue, 14 May 2024 09:58:40 UTC
log: 
  - status: requested
    timestamp: '2024-05-14T09:58:35.405000Z'
  - status: scheduled
    timestamp: '2024-05-14T09:58:35.568000Z'
  - status: running
    timestamp: '2024-05-14T09:58:35.619000Z'
  - status: failed
    timestamp: '2024-05-14T09:58:40.038000Z'
result: 
  errors:
  - message: Execution failed. See result for details.
    result:
      exit_code: 1
      result: None
      stderr: "st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "rabbitmq" (None)
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py", line 395, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py", line 214, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 93, in run
    setup_pack_virtualenv(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/util/virtualenvs.py", line 95, in setup_pack_virtualenv
    raise Exception(msg)
Exception: Pack "rabbitmq" is not installed. Looked in: /opt/stackstorm/packs
"
      stdout: ''
    task_id: install_pack_requirements
    type: error
  output:
    conflict_list: []
    message: ''
    packs_list:
    - rabbitmq
    warning_list: null
+--------------------------+------------------------+-----------------------------+-----------------------------+-------------------------------+
| id                       | status                 | task                        | action                      | start_timestamp               |
+--------------------------+------------------------+-----------------------------+-----------------------------+-------------------------------+
| 664335cb594a9b7ac2e9a836 | succeeded (1s elapsed) | init_task                   | core.noop                   | Tue, 14 May 2024 09:58:35 UTC |
| 664335cce6edd4837a15fa30 | succeeded (1s elapsed) | download_pack               | packs.download              | Tue, 14 May 2024 09:58:36 UTC |
| 664335cee6edd4837a15fa3f | succeeded (0s elapsed) | make_a_prerun               | packs.virtualenv_prerun     | Tue, 14 May 2024 09:58:38 UTC |
| 664335cee6edd4837a15fa4e | succeeded (0s elapsed) | get_pack_dependencies       | packs.get_pack_dependencies | Tue, 14 May 2024 09:58:38 UTC |
| 664335cfe6edd4837a15fa5d | succeeded (1s elapsed) | check_dependency_and_confli | core.noop                   | Tue, 14 May 2024 09:58:38 UTC |
|                          |                        | ct_list                     |                             |                               |
| 664335cfe6edd4837a15fa6c | failed (0s elapsed)    | install_pack_requirements   | packs.setup_virtualenv      | Tue, 14 May 2024 09:58:39 UTC |
+--------------------------+------------------------+-----------------------------+-----------------------------+-------------------------------+

2 case: with register_pack:

microk8s kubectl exec -it ${ST2CLIENT} --namespace default -- st2 pack install rabbitmq
Defaulted container "st2client" out of: st2client, generate-st2client-config (init)

For the "rabbitmq" pack, the following content will be registered:

actions   |  3
rules     |  0
sensors   |  1
aliases   |  0
triggers  |  0

Installation may take a while for packs with many items.

	[ succeeded ] init_task
	[ succeeded ] download_pack
	[ succeeded ] make_a_prerun
	[ succeeded ] get_pack_dependencies
	[ succeeded ] check_dependency_and_conflict_list
	[ succeeded ] install_pack_requirements
	[ succeeded ] get_pack_warnings
	[  failed   ] register_pack

id: 66433d4d2d96ea37337dd982
action.ref: packs.install
parameters: 
  packs:
  - rabbitmq
status: failed
start_timestamp: Tue, 14 May 2024 10:30:37 UTC
end_timestamp: Tue, 14 May 2024 10:30:45 UTC
log: 
  - status: requested
    timestamp: '2024-05-14T10:30:37.492000Z'
  - status: scheduled
    timestamp: '2024-05-14T10:30:37.614000Z'
  - status: running
    timestamp: '2024-05-14T10:30:37.661000Z'
  - status: failed
    timestamp: '2024-05-14T10:30:45.076000Z'
result: 
  errors:
  - message: Execution failed. See result for details.
    result:
      exit_code: 1
      result: None
      stderr: "st2.actions.python.St2RegisterAction: DEBUG    Calling client method "register" with kwargs "{'types': ['all'], 'packs': ['rabbitmq']}"
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py", line 395, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py", line 214, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py", line 76, in run
    result = self._run_client_method(
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py", line 155, in _run_client_method
    result = method(**method_kwargs)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py", line 45, in decorate
    return func(*args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py", line 684, in register
    self.handle_error(response)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py", line 218, in handle_error
    response.raise_for_status()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request
MESSAGE: Pack "rabbitmq" not found on disk: Directory "/opt/stackstorm/packs/rabbitmq" doesn't exist for url: http://stackstorm-ha-1715676993-st2api:9101/v1/packs/register
"
      stdout: ''
    task_id: register_pack
    type: error
  output:
    conflict_list: []
    message: Successfully installed packs
    packs_list:
    - rabbitmq
    warning_list: []
+--------------------------+------------------------+-----------------------------+-----------------------------+-------------------------------+
| id                       | status                 | task                        | action                      | start_timestamp               |
+--------------------------+------------------------+-----------------------------+-----------------------------+-------------------------------+
| 66433d4de6edd4837a15fd8f | succeeded (1s elapsed) | init_task                   | core.noop                   | Tue, 14 May 2024 10:30:37 UTC |
| 66433d4e594a9b7ac2e9ab87 | succeeded (1s elapsed) | download_pack               | packs.download              | Tue, 14 May 2024 10:30:38 UTC |
| 66433d4f594a9b7ac2e9ab96 | succeeded (1s elapsed) | make_a_prerun               | packs.virtualenv_prerun     | Tue, 14 May 2024 10:30:39 UTC |
| 66433d50594a9b7ac2e9aba5 | succeeded (0s elapsed) | get_pack_dependencies       | packs.get_pack_dependencies | Tue, 14 May 2024 10:30:40 UTC |
| 66433d51594a9b7ac2e9abb4 | succeeded (0s elapsed) | check_dependency_and_confli | core.noop                   | Tue, 14 May 2024 10:30:41 UTC |
|                          |                        | ct_list                     |                             |                               |
| 66433d51594a9b7ac2e9abc3 | succeeded (2s elapsed) | install_pack_requirements   | packs.setup_virtualenv      | Tue, 14 May 2024 10:30:41 UTC |
| 66433d53594a9b7ac2e9abd2 | succeeded (1s elapsed) | get_pack_warnings           | packs.get_pack_warnings     | Tue, 14 May 2024 10:30:43 UTC |
| 66433d54594a9b7ac2e9abe1 | failed (0s elapsed)    | register_pack               | packs.load                  | Tue, 14 May 2024 10:30:44 UTC |
+--------------------------+------------------------+-----------------------------+-----------------------------+-------------------------------+

with web interface:

{
  "output": {
    "packs_list": [
      "rabbitmq"
    ],
    "message": "Successfully installed packs",
    "conflict_list": [],
    "warning_list": []
  },
  "errors": [
    {
      "type": "error",
      "message": "Execution failed. See result for details.",
      "task_id": "register_pack",
      "result": {
        "stdout": "",
        "stderr": "st2.actions.python.St2RegisterAction: DEBUG    Calling client method \"register\" with kwargs \"{'types': ['all'], 'packs': ['rabbitmq']}\"\nTraceback (most recent call last):\n  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 395, in <module>\n    obj.run()\n  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 214, in run\n    output = action.run(**self._parameters)\n  File \"/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py\", line 76, in run\n    result = self._run_client_method(\n  File \"/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py\", line 155, in _run_client_method\n    result = method(**method_kwargs)\n  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 45, in decorate\n    return func(*args, **kwargs)\n  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 684, in register\n    self.handle_error(response)\n  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 218, in handle_error\n    response.raise_for_status()\n  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/models.py\", line 943, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 400 Client Error: Bad Request\nMESSAGE: Pack \"rabbitmq\" not found on disk: Directory \"/opt/stackstorm/packs/rabbitmq\" doesn't exist for url: http://stackstorm-ha-1715676993-st2api:9101/v1/packs/register\n",
        "exit_code": 1,
        "result": "None"
      }
    }
  ]
}

Output status pods:

kubectl get pods
NAME                                                           READY   STATUS      RESTARTS       AGE
mysql-deployment-67cc447bc8-9b2vc                              1/1     Running     5 (159m ago)   45h
stackstorm-ha-1715676993-job-st2-apikey-load-rdl9x             0/1     Completed   0              110m
stackstorm-ha-1715676993-job-st2-key-load-54ww4                0/1     Completed   0              109m
stackstorm-ha-1715676993-job-st2-register-content-5zwql        0/1     Completed   0              109m
stackstorm-ha-1715676993-mongodb-0                             1/1     Running     0              110m
stackstorm-ha-1715676993-mongodb-1                             1/1     Running     0              110m
stackstorm-ha-1715676993-mongodb-2                             1/1     Running     0              109m
stackstorm-ha-1715676993-rabbitmq-0                            1/1     Running     0              110m
stackstorm-ha-1715676993-rabbitmq-1                            1/1     Running     0              109m
stackstorm-ha-1715676993-rabbitmq-2                            1/1     Running     0              109m
stackstorm-ha-1715676993-redis-node-0                          2/2     Running     0              110m
stackstorm-ha-1715676993-redis-node-1                          2/2     Running     0              110m
stackstorm-ha-1715676993-redis-node-2                          2/2     Running     0              110m
stackstorm-ha-1715676993-st2actionrunner-5c54d8f69f-72hjt      1/1     Running     0              110m
stackstorm-ha-1715676993-st2actionrunner-5c54d8f69f-8pj2j      1/1     Running     0              110m
stackstorm-ha-1715676993-st2actionrunner-5c54d8f69f-9dhv8      1/1     Running     0              110m
stackstorm-ha-1715676993-st2actionrunner-5c54d8f69f-gwnpm      1/1     Running     0              110m
stackstorm-ha-1715676993-st2actionrunner-5c54d8f69f-pzmj7      1/1     Running     0              110m
stackstorm-ha-1715676993-st2api-6b8459dbf9-hfp69               1/1     Running     0              110m
stackstorm-ha-1715676993-st2api-6b8459dbf9-qqqhv               1/1     Running     0              110m
stackstorm-ha-1715676993-st2auth-7b5987c99f-572fc              1/1     Running     0              110m
stackstorm-ha-1715676993-st2auth-7b5987c99f-6vbcs              1/1     Running     0              110m
stackstorm-ha-1715676993-st2client-7595ccbd6-hnk89             1/1     Running     0              110m
stackstorm-ha-1715676993-st2garbagecollector-6d66c67d4-xshxc   1/1     Running     0              110m
stackstorm-ha-1715676993-st2notifier-644696c5dc-6rkhb          1/1     Running     0              110m
stackstorm-ha-1715676993-st2notifier-644696c5dc-r7mjm          1/1     Running     0              110m
stackstorm-ha-1715676993-st2rulesengine-6f8bcf6dcd-wktjz       1/1     Running     0              110m
stackstorm-ha-1715676993-st2rulesengine-6f8bcf6dcd-xmlv6       1/1     Running     0              110m
stackstorm-ha-1715676993-st2scheduler-f6c5cf889-jzn22          1/1     Running     0              110m
stackstorm-ha-1715676993-st2scheduler-f6c5cf889-xk7jm          1/1     Running     0              110m
stackstorm-ha-1715676993-st2sensorcontainer-54b8689df-plg6r    1/1     Running     0              110m
stackstorm-ha-1715676993-st2stream-79f5b67b5d-z5jph            1/1     Running     0              110m
stackstorm-ha-1715676993-st2stream-79f5b67b5d-zjw75            1/1     Running     0              110m
stackstorm-ha-1715676993-st2timersengine-77c8c8c86c-8g7kx      1/1     Running     0              110m
stackstorm-ha-1715676993-st2web-58f7b6c5c4-f6qj4               1/1     Running     0              110m
stackstorm-ha-1715676993-st2web-58f7b6c5c4-rz888               1/1     Running     0              110m
stackstorm-ha-1715676993-st2workflowengine-7989b8d74f-hjmgn    1/1     Running     0              110m
stackstorm-ha-1715676993-st2workflowengine-7989b8d74f-t226r    1/1     Running     0              110m

[zsmanjot]

If this may help you .

#5880

[chain312]

Where should your application package be installed, and whether you use PVC or NFS to store the package
Install custom st2 packs in the cluster