st2actionrunner talk to redis question

[guzzijones]

how does the st2actionrunner know what ip and port the redis server is running on?
I would expect some sort of environment variable passed into the action runner container but I don't see that.
Is the redis port somehow mapped to all pods?

[fdrab]

2024-07-01 11:07:48,299 INFO [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
/etc/st2/st2.docker.conf has:
[coordination]
url = redis://@dev-redis-node-0.dev-redis-headless.stackstorm.svc.cluster.local:26379?sentinel=mymaster&sentinel_fallback=dev-redis-node-1.dev-redis-headless.stackstorm.svc.cluster.local:26379

[crazykingit]

thanks @fdrab for your input. the st2 services are having problems locating the redis master. any help you or other people can provide is appreciated.

error message for st2api. my st2scheduler has the similar error message of not able to locate redis master

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/cmd/api.py", line 99, in main
    return _run_server()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/cmd/api.py", line 87, in _run_server
    sock, app.setup_app(), custom_pool=worker_pool, log=LOG, log_output=False
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/app.py", line 84, in setup_app
    router.add_spec(spec, transforms=transforms)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/router.py", line 262, in add_spec
    __import__(module_name)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/controllers/v1/keyvalue.py", line 533, in <module>
    key_value_pair_controller = KeyValuePairController()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/controllers/v1/keyvalue.py", line 63, in __init__
    self._coordinator = coordination.get_coordinator()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/services/coordination.py", line 253, in get_coordinator
    COORDINATOR = coordinator_setup(start_heart=start_heart)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/services/coordination.py", line 224, in coordinator_setup
    coordinator.start(start_heart=start_heart)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/coordination.py", line 689, in start
    super(CoordinationDriverWithExecutor, self).start(start_heart)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/coordination.py", line 426, in start
    self._start()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 450, in _start
    self._server_info = self._client.info()
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 44, in _translate_failures
    utils.raise_with_cause(coordination.ToozConnectionError,
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/utils.py", line 224, in raise_with_cause
    excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_utils/excutils.py", line 142, in raise_with_cause
    raise exc_cls(message, *args, **kwargs) from kwargs.get('cause')
tooz.coordination.ToozConnectionError: No master found for 'master'

here is my setup for redis
st2 conf file:

    [coordination]
    url = redis://default:[email protected]:26379?sentinel=master

redis values.yaml

  ## @param sentinel.masterSet Master set name
  ##
  masterSet: master

in st2client pod, using redis-cli, i was able to locate my redis master

root@stackstorm-st2client-bb9b4c57c-zt7z7:/opt/stackstorm# redis-cli -h redis-headless.redis-dev.svc.cluster.local -p 26379

redis-headless.redis-dev.svc.cluster.local:26379> auth password
OK

redis-headless.redis-dev.svc.cluster.local:26379> sentinel masters
1)  1) "name"
    2) "master"
    3) "ip"
    4) "redis-node-0.redis-headless.redis-dev.svc.cluster.local"
    5) "port"
    6) "6379"

[fdrab]

can't say I'm an expert on any of this, what does your redis section of the values.yaml look like?

[fdrab]

did you solve this?

[fdrab]

For anyone facing this, set the redis.sentinel.usePassword to false explicitly. the default value is true.

Also avoid having "=" in password (maybe some other special chars as well) as they get url encoded.

[crazykingit]

did you solve this?

no. i switched my redis setup to a master-secondary mode instead of using the sentinel mode. this got me going for now.
i have redis, rabbitmq, and mangodb running in their own namespaces