st2actionrunner talk to redis question #423

Activity

1. fdrab commented on Jul 1, 2024

   

   fdrab

   on Jul 1, 2024

   Last edited by fdrab

   More actions

   2024-07-01 11:07:48,299 INFO [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
   /etc/st2/st2.docker.conf has:
   [coordination]
   url = redis://@dev-redis-node-0.dev-redis-headless.stackstorm.svc.cluster.local:26379?sentinel=mymaster&sentinel_fallback=dev-redis-node-1.dev-redis-headless.stackstorm.svc.cluster.local:26379
2. crazykingit commented on Jul 2, 2024

   

   crazykingit

   on Jul 2, 2024

   Last edited by crazykingit

   More actions

   thanks @fdrab for your input. the st2 services are having problems locating the redis master. any help you or other people can provide is appreciated.

   error message for st2api. my st2scheduler has the similar error message of not able to locate redis master

   Traceback (most recent call last):
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/cmd/api.py", line 99, in main
       return _run_server()
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/cmd/api.py", line 87, in _run_server
       sock, app.setup_app(), custom_pool=worker_pool, log=LOG, log_output=False
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/app.py", line 84, in setup_app
       router.add_spec(spec, transforms=transforms)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/router.py", line 262, in add_spec
       __import__(module_name)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/controllers/v1/keyvalue.py", line 533, in <module>
       key_value_pair_controller = KeyValuePairController()
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/controllers/v1/keyvalue.py", line 63, in __init__
       self._coordinator = coordination.get_coordinator()
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/services/coordination.py", line 253, in get_coordinator
       COORDINATOR = coordinator_setup(start_heart=start_heart)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/services/coordination.py", line 224, in coordinator_setup
       coordinator.start(start_heart=start_heart)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/coordination.py", line 689, in start
       super(CoordinationDriverWithExecutor, self).start(start_heart)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/coordination.py", line 426, in start
       self._start()
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 450, in _start
       self._server_info = self._client.info()
     File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
       self.gen.throw(type, value, traceback)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 44, in _translate_failures
       utils.raise_with_cause(coordination.ToozConnectionError,
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/utils.py", line 224, in raise_with_cause
       excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
     File "/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_utils/excutils.py", line 142, in raise_with_cause
       raise exc_cls(message, *args, **kwargs) from kwargs.get('cause')
   tooz.coordination.ToozConnectionError: No master found for 'master'
   

   here is my setup for redis
   st2 conf file:

       [coordination]
       url = redis://default:password@redis-headless.redis-dev.svc.cluster.local:26379?sentinel=master
   

   redis values.yaml

     ## @param sentinel.masterSet Master set name
     ##
     masterSet: master
   
   

   in st2client pod, using redis-cli, i was able to locate my redis master

   root@stackstorm-st2client-bb9b4c57c-zt7z7:/opt/stackstorm# redis-cli -h redis-headless.redis-dev.svc.cluster.local -p 26379
   
   redis-headless.redis-dev.svc.cluster.local:26379> auth password
   OK
   
   redis-headless.redis-dev.svc.cluster.local:26379> sentinel masters
   1)  1) "name"
       2) "master"
       3) "ip"
       4) "redis-node-0.redis-headless.redis-dev.svc.cluster.local"
       5) "port"
       6) "6379"
   
   
3. fdrab commented on Jul 4, 2024

   

   fdrab

   on Jul 4, 2024

   More actions

   can't say I'm an expert on any of this, what does your redis section of the values.yaml look like?
4. fdrab commented on Jul 15, 2024

   

   fdrab

   on Jul 15, 2024

   More actions

   did you solve this?
5. fdrab commented on Jul 15, 2024

   

   fdrab

   on Jul 15, 2024

   More actions

   For anyone facing this, set the redis.sentinel.usePassword to false explicitly. the default value is true.

   Also avoid having "=" in password (maybe some other special chars as well) as they get url encoded.
6. crazykingit commented on Jul 16, 2024

   

   crazykingit

   on Jul 16, 2024

   More actions

   did you solve this?

   no. i switched my redis setup to a master-secondary mode instead of using the sentinel mode. this got me going for now.
   i have redis, rabbitmq, and mangodb running in their own namespaces