PgBackrest: unable to create stanza

[henryx]

Report

Pgbackrest cannot perform a backup to S3 because the endpoint address is generated in a wrong way

More about the problem

Reported logs is here:

2024-05-07T19:04:54.455Z	ERROR	unable to create stanza	{"controller": "postgrescluster", "controllerGroup": "postgres-operator.crunchydata.com", "controllerKind": "PostgresCluster", "PostgresCluster": {"name":"fap-cluster-pg-db","namespace":"postgres"}, "namespace": "postgres", "name": "fap-cluster-pg-db", "reconcileID": "4b48a586-101e-492c-b832-12cd2eb6c17d", "reconciler": "pgBackRest", "error": "command terminated with exit code 49: ERROR: [049]: unable to get address for 'postgres.minio-api.domain.com': [-2] Name or service not known\n", "errorVerbose": "command terminated with exit code 49: ERROR: [049]: unable to get address for 'postgres.minio-api.domain.com': [-2] Name or service not known\n\ngithub.com/percona/percona-postgresql-operator/internal/pgbackrest.Executor.StanzaCreateOrUpgrade\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/pgbackrest/pgbackrest.go:96\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcileStanzaCreate\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:2650\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcilePGBackRest\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:1360\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:356\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcileStanzaCreate\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:2657\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcilePGBackRest\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:1360\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:356\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}

Steps to reproduce

1. Create a secret with S3 credentials

apiVersion: v1
kind: Secret
metadata:
  name: a-cluster-pg-db-pgbackrest-secrets
type: Opaque
stringData:
  s3.conf: |
    [global]
    repo1-s3-key=a_key
    repo1-s3-key-secret=a_secret
    repo1-storage-verify-tls=n
EOF

2. Create a cluster instance via Helm specifying a custom S3 endpoint:

backups:
  pgbackrest:
    configuration:
    - secret:
        name: fap-cluster-pg-db-pgbackrest-secrets
    repos:
    - name: repo1
      schedules:
        full: "0 * * * *"
      s3:
        bucket: "postgres"
        endpoint: "https://minio-api.domain.com/"
        region: custom

Versions

1. Kubernetes: 1.28.9+rke2r1
2. Operator: 2.3.1
3. Database: 16

Anything else?

According to the logs, it seems that S3 endpoint is constructed as {bucket}.{endpoint} instead of {endpoint}/{bucket}

[discostur]

@henryx the documentation says

The repo2-s3-uri-style option [should be set to path ](https://pgbackrest.org/configuration.html#section-repository/option-repo-s3-uri-style) if you use S3-compatible storage (otherwise you might see “host not found error” in your backup job logs), and is not needed for Amazon S3.

https://docs.percona.com/percona-operator-for-postgresql/2.0/backups-storage.html

[nick2432]

can i work on this?