Redis tls not working (certificate verify failed)

[NicosKaralis]

Which chart: redis:16.2.1

Describe the bug

I generated some certs with easyRSA, the same way I have with nginx and PostgreSQL.
I've tried locally to run redis-server and redis-cli with the tls certs and it works fine.

I added the ca.cert, server.key and server.cert to a secret and just to make sure, I ran again redis server and client with the kubernetes secrets using kubectl get secret redis-tls -n redis --output="jsonpath={ .data.ca\.crt }" | base64 -d just to make sure the secrets are ok and it worked

After I install the chart and the container it 0/1 Running I can open the logs and see these lines:

Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed

I disabled the probes so the container is not restarted all the time and the issue still persists, I can use the server keys or the client keys, nothing changes

To Reproduce

Steps to reproduce the behavior:

1. easyrsa build-ca
2. easyrsa build-server-full redis-server nopass
3. easyrsa build-client-full redis-client nopass
4. Add the ca.crt, server.key and server.crt to a kubernetes secret (doesn't matter if opaque or tls secret)
5. Set values.yaml

architecture: standalone
tls:
  enabled: true
  # autoGenerated: true
  # certificatesSecret: redis-tls
  existingSecret: redis-tls
  certFilename: tls.crt
  certKeyFilename: tls.key
  certCAFilename: ca.crt
  dhParamsFilename: dh.pem

6. Run helm install my-redis bitnami/redis -n redis -f values/redis.yaml
7. Run kubectl logs -f pod/$(kubectl get pods -A --template='{{range .items}}{{ .metadata.name }}{{" -n "}}{{ .metadata.namespace }}{{"\n"}}{{end}}' | grep redis)
8. See the error logs
9. Disable probes in values.yml

architecture: standalone
tls:
  enabled: true
  # autoGenerated: true
  # certificatesSecret: redis-tls
  existingSecret: redis-tls
  certFilename: tls.crt
  certKeyFilename: tls.key
  certCAFilename: ca.crt
  dhParamsFilename: dh.pem
master:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false

10. Run helm install my-redis bitnami/redis -n redis -f values/redis.yaml
11. Run

redis-cli -h localhost --tls \
  -p $(kubectl get service redis-master -n redis --output="jsonpath={ .spec.ports[0].nodePort }") \
  --cacert <(kubectl get secret redis-tls -n redis --output="jsonpath={ .data.ca\.crt }" | base64 -d) \
  --cert <(kubectl get secret redis-tls -n redis --output="jsonpath={ .data.tls\.crt }" | base64 -d) \
  --key <(kubectl get secret redis-tls -n redis --output="jsonpath={ .data.tls\.key }" | base64 -d) \
  ping

12. See the error logs

Expected behavior

The certs that work on my machine should also work on the container

Additional context

Log when probes are enabled:

1:C 01 Feb 2022 10:11:57.390 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 01 Feb 2022 10:11:57.390 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 01 Feb 2022 10:11:57.390 # Configuration loaded
1:M 01 Feb 2022 10:11:57.391 * monotonic clock: POSIX clock_gettime
1:M 01 Feb 2022 10:11:57.391 * Running mode=standalone, port=6379.
1:M 01 Feb 2022 10:11:57.391 # Server initialized
1:M 01 Feb 2022 10:11:57.392 * Ready to accept connections
1:M 01 Feb 2022 10:12:21.130 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:21.253 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed (conn: fd=9)
1:M 01 Feb 2022 10:12:26.114 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:26.183 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:31.116 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:31.175 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:36.118 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:36.223 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:41.114 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:M 01 Feb 2022 10:12:41.232 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
1:signal-handler (1643710361) Received SIGTERM scheduling shutdown...
1:M 01 Feb 2022 10:12:41.321 # User requested shutdown...
1:M 01 Feb 2022 10:12:41.321 * Calling fsync() on the AOF file.
1:M 01 Feb 2022 10:12:41.321 # Redis is now ready to exit, bye bye...

Logs when probe is not enabled

1:C 01 Feb 2022 10:17:27.661 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 01 Feb 2022 10:17:27.661 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 01 Feb 2022 10:17:27.661 # Configuration loaded
1:M 01 Feb 2022 10:17:27.662 * monotonic clock: POSIX clock_gettime
1:M 01 Feb 2022 10:17:27.662 * Running mode=standalone, port=6379.
1:M 01 Feb 2022 10:17:27.662 # Server initialized
1:M 01 Feb 2022 10:17:27.663 * Ready to accept connections
1:M 01 Feb 2022 10:17:57.259 # Error accepting a client connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed

[juan131]

Hi @NicosKaralis

Are you able to reproduce using using auto-generated certs (tls.autoGenerated=true) or only when using your custom certs?

[NicosKaralis]

Auto generated certs work fine, mine don't

I'm using the same procedure to generate nginx client certs and PostgreSQL certs, both are working

Also, on my machine it works, the certs are accepted, but on the container, they give this error

[juan131]

Hi @NicosKaralis

Did you configure the SAN (Subject Alternative Names) including Redis service DNS name when creating the certificates? Check how's it done using auto-generated certificates:

• https://github.com/bitnami/charts/blob/master/bitnami/redis/templates/tls-secret.yaml#L8-L9
• https://helm.sh/docs/chart_template_guide/function_list/#genselfsignedcert

This is usually required if the clients validate the hostname in the certificate emitted by the server.

[NicosKaralis]

I'm using easyRSA to generate the certificates, they have DNS related attributes

Also, I'm using the same client (redis-cli 6.2.6) from my machine to connect to the container and the local redis-server

Just to try make things clearer:

• Auto generated certs works fine
• Manually generated certs works on local redis-server but does not work inside container

The liveness_probe can't success and the container is killed all the time

[juan131]

I'm using easyRSA to generate the certificates, they have DNS related attributes

Are you including "localhost" and "127.0.0.1" in the SAN? Maybe the reason why probes are failing is because you're missing that hostname and therefore the client can't verify the certificates. Please check the list of alternatives names we set here:

• https://github.com/bitnami/charts/blob/master/bitnami/redis/templates/tls-secret.yaml#L8-L9

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[sylguyot]

I have exactly the same problem with certificates (stored in a secret) generated by cert-manager
whereas it works if I generate certificates with openssl , then the secret:

curl -sSlO https://raw.githubusercontent.com/redis/redis/unstable/utils/gen-test-certs.sh
chmod +x gen-test-certs.sh
./gen-test-certs.sh
 kubectl create secret generic certificates-tls-secret --from-file=tls.crt=./tests/tls/redis.crt --from-file=tls.key=./tests/tls/redis.key --from-file=ca.crt=./tests/tls/ca.crt -n redis-ns

[Jamison1]

I had the same error as the OP and the search for an answer led me here so posting how I got past the error in hopes it helps someone else.

I generate my own Intermediate Cert for the dev cluster and use cert-manager to generate the server cert and key.

I had the same error so I tested the auto generated cert with redis and everything matched except for the extended usage param.

I needed to add a usage param client auth to the Certificate file for cert-manager in addition to the other 3 I had:

usages:
  - digital signature
  - key encipherment
  - server auth
  - client auth

After adding client auth, then the liveness and readiness probe worked fine with no restarts so far.

In looking into the usages params with cert-manager, it appears it doesn't add client auth if not specified.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[FedeMarchiniHotovo]

@juan131 Hey Juan, how does autogenerated tls work? I just enabled it in helm chart and then don't understand how to proceed. Documentation is not too much clear for me.

[juan131]

Hi @FedeMarchiniHotovo

TLS certs are created using Helm capabilities (see genCA and genSignedCert https://masterminds.github.io/sprig/crypto.html), included in K8s secrets and mounted as read-only files in Redis containers. Find below the manifest where this autogeneration happens:

• https://github.com/bitnami/charts/blob/main/bitnami/redis/templates/tls-secret.yaml

[FedeMarchiniHotovo]

@juan131 Those cert files are ready to be used by software clients like redis-cli to connect with redis server in this way redis-cli -h redis-master.redis.svc.cluster.local --tls --cert redis.crt --key redis.key --cacert ca.crt -p 6379 or do I have to first do some configuration with them from redis server side?

[FedeMarchiniHotovo]

Think I got the solution ! Thank u so much for the help

[tdg5]

For others who run into this problem I suggest looking at #30305 which may offer a workaround in some cases, particularly if you're trying to use a cert-manager certificate that doesn't include ca.crt.

Also, in my experience,

including "localhost" and "127.0.0.1" in the SAN

is not necessary and not possible unless certificates are self-signed, so if you're using a real certificate, don't bother with this path.