Retained messages being lost after pod restart even after using Persistent volume

[ManishJain-TomTom]

We're running a cluster with 2 VerneMq brokers. Everything works fine if we start one broker at one time, but as soon as we need to re-start both the brokers, all the retained messages are lost.

To fix the issue, we tried configuring persistent volume with vernemq, and we can see that the claim is bound to vernemq and volume is created. Even after this when we tested our scenario by restarting both the pods, we found that the retained messages were not synced. This is leading to data loss.

Below is the configuration we are using to create storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-storage
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Standard_LRS
  kind: managed
reclaimPolicy: Retain

Our vernemq configuration looks like this:

{{- if .Values.persistentVolume.enabled }}
  volumeClaimTemplates:
    - metadata:
        name: data
        annotations:
        {{- range $key, $value := .Values.persistentVolume.annotations }}
          {{ $key }}: {{ $value }}
        {{- end }}
      spec:
        accessModes:
        {{- range .Values.persistentVolume.accessModes }}
          - {{ . | quote }}
        {{- end }}
        resources:
          requests:
            storage: {{ .Values.persistentVolume.size }}
      {{- if .Values.persistentVolume.storageClass }}
      {{- if (eq "-" .Values.persistentVolume.storageClass) }}
        storageClassName: ""
      {{- else }}
        storageClassName: "{{ .Values.persistentVolume.storageClass }}"
      {{- end }}
      {{- end }}
{{- else }}
        - name: data
{{- end }}

PVCs are properly treated. Kubectl get pvc returns us the created PVCs:

data-vernemq-0             Bound    xxx   5Gi        RWO            default        19h
data-vernemq-1             Bound    xxx   5Gi        RWO            default        18h

Storage class:

NAME                    PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
azure-storage      kubernetes.io/azure-disk  Retain                      Immediate                                                      false                  19h

Is there something that I am missing in configuration?

Environment

• VerneMQ Version: 1.12.4
• Cluster size/standalone: 2 brokers

Expected behaviour

Since persistent volume is being configured, vernemq nodes should be able to sync the retained messages on restart, and there should be no data loss.

Actual behaviour

Retained messages are not synced after both pods are restarted.

[ManishJain-TomTom]

Hi Team, can someone please assist me on this issue? Or point me to documentation on how to configure the persistent volumes?

[ioolkos]

@ManishJain-TomTom Thanks, do you have subscriptions and/or support agreement in place? if not, please contact over "contact @ octavolabs ".

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[ioolkos]

Closed in favour of: vernemq/docker-vernemq#326

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.