Skip to content

Commit

Permalink
Resolve saiem bhai's review changes
Browse files Browse the repository at this point in the history
Signed-off-by: Anisur Rahman <[email protected]>
  • Loading branch information
anisurrahman75 authored and tamalsaha committed Sep 20, 2024
1 parent 7382645 commit bd72a71
Show file tree
Hide file tree
Showing 14 changed files with 85 additions and 39 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ spec:
allowedNamespaces:
from: All
default: true
deletionPolicy: WipeOut
deletionPolicy: Delete
11 changes: 5 additions & 6 deletions docs/guides/postgres/backup/kubestash/application-level/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ spec:
allowedNamespaces:
from: All
default: true
deletionPolicy: WipeOut
deletionPolicy: Delete
```

Let's create the BackupStorage we have shown above,
Expand Down Expand Up @@ -425,8 +425,7 @@ spec:

- `.spec.sessions[*].schedule` specifies that we want to backup at `5 minutes` interval.
- `.spec.target` refers to the targeted `sample-postgres` PostgreSQL database that we created earlier.
- `.spec.sessions[*].addon.tasks[*].name[*]` specifies that both the `manifest-backup` and `logical-backup` will be taken together.

- `.spec.sessions[*].addon.tasks[*].name[*]` specifies that both the `manifest-backup` and `logical-backup` tasks will be executed.

Let's create the `BackupConfiguration` CR that we have shown above,

Expand Down Expand Up @@ -484,9 +483,9 @@ We can see from the above output that the backup session has succeeded. Now, we
Once a backup is complete, KubeStash will update the respective `Repository` CR to reflect the backup. Check that the repository `sample-postgres-backup` has been updated by the following command,

```bash
$ kubectl get repository -n demo sample-postgres-backup
$ kubectl get repository -n demo gcs-postgres-repo
NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE
sample-postgres-backup true 1 806 B Ready 8m27s 9m18s
gcs-postgres-repo true 1 806 B Ready 8m27s 9m18s
```

At this moment we have one `Snapshot`. Run the following command to check the respective `Snapshot` which represents the state of a backup run for an application.
Expand Down Expand Up @@ -594,7 +593,7 @@ status:
totalComponents: 2
```

> KubeStash uses a logical backup approach to take backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`. Do the same for auto-backup, application backup and customize backup if necessary.
> KubeStash uses `pg_dump` or `pg_dumpall` to perform backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`.

> KubeStash set component name as `manifest` for the `manifest backup` of PostgreSQL databases.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ spec:
allowedNamespaces:
from: All
default: true
deletionPolicy: WipeOut
deletionPolicy: Delete
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ spec:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
# ============== Blueprint for Backends of BackupConfiguration =================
backends:
- name: gcs-backend
storageRef:
Expand All @@ -18,7 +17,6 @@ spec:
retentionPolicy:
name: demo-retention
namespace: demo
# ============== Blueprint for Sessions of BackupConfiguration =================
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ spec:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
# ============== Blueprint for Backends of BackupConfiguration =================
backends:
- name: gcs-backend
storageRef:
Expand All @@ -18,7 +17,6 @@ spec:
retentionPolicy:
name: demo-retention
namespace: demo
# ============== Blueprint for Sessions of BackupConfiguration =================
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
Expand Down
11 changes: 3 additions & 8 deletions docs/guides/postgres/backup/kubestash/auto-backup/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ spec:
allowedNamespaces:
from: All
default: true
deletionPolicy: WipeOut
deletionPolicy: Delete
```
Let's create the BackupStorage we have shown above,
Expand Down Expand Up @@ -359,7 +359,6 @@ status:

Notice the `spec.backends`, `spec.sessions` and `spec.target` sections, KubeStash automatically resolved those info from the `BackupBluePrint` and created above `BackupConfiguration`.


**Verify BackupSession:**

KubeStash triggers an instant backup as soon as the `BackupConfiguration` is ready. After that, backups are scheduled according to the specified schedule.
Expand Down Expand Up @@ -475,13 +474,12 @@ status:
totalComponents: 1
```

> KubeStash uses a logical backup approach to take backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`. Do the same for auto-backup, application backup and customize backup if necessary.
> KubeStash uses `pg_dump` or `pg_dumpall` to perform backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`.

Now, if we navigate to the GCS bucket, we will see the backed up data stored in the `blueprint/default-blueprint/repository/v1/frequent-backup/dump` directory. KubeStash also keeps the backup for `Snapshot` YAMLs, which can be found in the `blueprint/default-blueprint/snapshots` directory.

> Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.


## Auto-backup with custom configurations

In this section, we are going to backup a `PostgreSQL` database of `demo` namespace. We are going to use the custom configurations which will be specified in the `BackupBlueprint` CR.
Expand All @@ -504,7 +502,6 @@ spec:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
# ============== Blueprint for Backends of BackupConfiguration =================
backends:
- name: gcs-backend
storageRef:
Expand All @@ -513,7 +510,6 @@ spec:
retentionPolicy:
name: demo-retention
namespace: demo
# ============== Blueprint for Sessions of BackupConfiguration =================
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
Expand Down Expand Up @@ -834,8 +830,7 @@ status:
totalComponents: 1
```


> KubeStash uses a logical backup approach to take backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`. Do the same for auto-backup, application backup and customize backup if necessary.
> KubeStash uses `pg_dump` or `pg_dumpall` to perform backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`.

Now, if we navigate to the GCS bucket, we will see the backed up data stored in the `blueprint/demo/sample-postgres-2/repository/v1/frequent-backup/dump` directory. KubeStash also keeps the backup for `Snapshot` YAMLs, which can be found in the `blueprint/demo/sample-postgres-2/snapshots` directory.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@ spec:
retentionPolicy:
name: demo-retention
namespace: demo
- name: s3-backend
storageRef:
namespace: demo
name: s3-storage
retentionPolicy:
name: demo-retention
namespace: demo
sessions:
- name: frequent-backup
scheduler:
Expand All @@ -30,8 +37,8 @@ spec:
encryptionSecret:
name: encrypt-secret
namespace: demo
- name: gcs-postgres-repo-2
backend: gcs-backend
- name: s3-postgres-repo
backend: s3-backend
directory: /postgres-copy
encryptionSecret:
name: encrypt-secret
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ spec:
usagePolicy:
allowedNamespaces:
from: All
default: true
default: false
deletionPolicy: WipeOut
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
apiVersion: storage.kubestash.com/v1alpha1
kind: BackupStorage
metadata:
name: s3-storage
namespace: demo
spec:
storage:
provider: s3
s3:
bucket: kubestash
region: us-east-1
endpoint: us-east-1.linodeobjects.com
secretName: s3-secret
prefix: sunny
usagePolicy:
allowedNamespaces:
from: All
default: false
deletionPolicy: WipeOut
17 changes: 12 additions & 5 deletions docs/guides/postgres/backup/kubestash/customization/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ spec:
### Passing a target database to the backup process
KubeStash PostgreSQL addon uses the [pg_dumpall](https://www.postgresql.org/docs/current/app-pg-dumpall.html) command by default for backups. For a single database backup, you need to rewrite the dump command. You can do this by setting `backupCmd` to [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) under the `addon.tasks[*].params` section and specifying the database name using the `args` parameter in the same section.
KubeStash PostgreSQL addon uses the [pg_dumpall](https://www.postgresql.org/docs/current/app-pg-dumpall.html) command by default for backups. If you want to back up a single database, you’ll need to switch the command to [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html). You can do this by setting `backupCmd` to `pg_dump` under the `addon.tasks[*].params` section and specifying the database name using the `args` parameter in the same section.

The below example shows how you can set `pg_dump` and pass target database name during backup.

Expand Down Expand Up @@ -117,9 +117,9 @@ spec:

> **WARNING**: Make sure that your provided database has been created before taking backup.

### Using multiple repositories
### Using multiple backends

You can configure multiple repositories for the same backend. For example, if you want to back up the `/postgres` directory using the `gcs-postgres-repo` repository, you can also back up another directory, such as `/postgres-copy`, by using a different repository, like `gcs-postgres-repo-2`.
You can configure multiple backends within a single `backupConfiguration`. To back up the same data to different backends, such as S3 and GCS, declare each backend in the `.spe.backends` section. Then, reference these backends in the `.spec.sessions[*].repositories` section.

```yaml
apiVersion: core.kubestash.com/v1alpha1
Expand All @@ -141,6 +141,13 @@ spec:
retentionPolicy:
name: demo-retention
namespace: demo
- name: s3-backend
storageRef:
namespace: demo
name: s3-storage
retentionPolicy:
name: demo-retention
namespace: demo
sessions:
- name: frequent-backup
scheduler:
Expand All @@ -154,8 +161,8 @@ spec:
encryptionSecret:
name: encrypt-secret
namespace: demo
- name: gcs-postgres-repo-2
backend: gcs-backend
- name: s3-postgres-repo
backend: s3-backend
directory: /postgres-copy
encryptionSecret:
name: encrypt-secret
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
apiVersion: core.kubestash.com/v1alpha1
kind: RestoreSession
metadata:
name: sample-postgres-restore
namespace: demo
spec:
target:
apiGroup: kubedb.com
kind: Postgres
namespace: demo
name: restored-postgres
dataSource:
repository: gcs-postgres-repo
snapshot: latest
encryptionSecret:
name: encrypt-secret
namespace: demo
addon:
name: postgres-addon
tasks:
- name: logical-backup-restore
params:
args: --dbname=testdb
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ spec:
allowedNamespaces:
from: All
default: true
deletionPolicy: WipeOut
deletionPolicy: Delete
8 changes: 4 additions & 4 deletions docs/guides/postgres/backup/kubestash/logical/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ spec:
allowedNamespaces:
from: All
default: true
deletionPolicy: WipeOut
deletionPolicy: Delete
```

Let's create the BackupStorage we have shown above,
Expand Down Expand Up @@ -483,9 +483,9 @@ We can see from the above output that the backup session has succeeded. Now, we
Once a backup is complete, KubeStash will update the respective `Repository` CR to reflect the backup. Check that the repository `sample-postgres-backup` has been updated by the following command,

```bash
$ kubectl get repository -n demo sample-postgres-backup
$ kubectl get repository -n demo gcs-postgres-repo
NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE
sample-postgres-backup true 1 806 B Ready 8m27s 9m18s
gcs-postgres-repo true 1 806 B Ready 8m27s 9m18s
```

At this moment we have one `Snapshot`. Run the following command to check the respective `Snapshot` which represents the state of a backup run for an application.
Expand Down Expand Up @@ -581,7 +581,7 @@ status:
totalComponents: 1
```

> KubeStash uses a logical backup approach to take backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`. Do the same for auto-backup, application backup and customize backup if necessary.
> KubeStash uses `pg_dump` or `pg_dumpall` to perform backups of target `PostgreSQL` databases. Therefore, the component name for logical backups is set as `dump`.

Now, if we navigate to the GCS bucket, we will see the backed up data stored in the `demo/popstgres/repository/v1/frequent-backup/dump` directory. KubeStash also keeps the backup for `Snapshot` YAMLs, which can be found in the `demo/postgres/snapshots` directory.

Expand Down
12 changes: 6 additions & 6 deletions docs/guides/postgres/backup/kubestash/overview/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@ The backup process consists of the following steps:

2. Then, she creates a `BackupStorage` custom resource that specifies the backend information, along with the `Secret` containing the credentials needed to access the backend.

3. KubeStash operator watches for `BackupStorage` custom resources. When it finds a `BackupStorage` object, it initializes the `BackupStorage` by uploading the `metadata.yaml` file into the target storage.
3. KubeStash operator watches for `BackupStorage` custom resources. When it finds a `BackupStorage` object, it initializes the `BackupStorage` by uploading the `metadata.yaml` file to the specified backend.

4. Then, she creates a `BackupConfiguration` custom resource that specifies the targeted the KubeDB managed `PostgreSQL` database, the `Addon` info with a specified task, etc. It also provides information about one or more repositories, each indicating a path and a `BackupStorage` for storing the backed-up data.
4. Next, she creates a `BackupConfiguration` custom resource that specifies the target database, addon information (including backup tasks), backup schedules, storage backends for storing the backup data, and other additional settings.

5. KubeStash operator watches for `BackupConfiguration` objects.

Expand All @@ -50,7 +50,7 @@ The backup process consists of the following steps:

8. Then, it creates a `CronJob` for each session with the schedule specified in `BackupConfiguration` to trigger backup periodically.

9. KubeStash operator triggers an instant backup as soon as the `BackupConfiguration` is ready. After that, backups are triggered by the `CronJob` according to the specified schedule.
9. KubeStash operator triggers an instant backup as soon as the `BackupConfiguration` is ready. Backups are otherwise triggered by the `CronJob` based on the specified schedule.

10. KubeStash operator watches for `BackupSession` custom resources.

Expand All @@ -60,7 +60,7 @@ The backup process consists of the following steps:

13. Then, it creates the `Job` to backup the targeted `PostgreSQL` database.

14. The backup `Job` reads necessary information (e.g. auth secret, port) to connect with the database from the `AppBinding` crd. It also reads backend information and access credentials from BackupStorage crd, Storage Secret and Repository path respectively.
14. The backup `Job` reads necessary information (e.g. auth secret, port) to connect with the database from the `AppBinding` CR. It also reads backend information and access credentials from `BackupStorage` CR, Storage Secret and `Repository` path respectively.

15. Then, the `Job` dumps the targeted `PostgreSQL` database and uploads the output to the backend. KubeStash pipes the output of dump command to uploading process. Hence, backup `Job` does not require a large volume to hold the entire dump output.

Expand All @@ -79,15 +79,15 @@ The restore process consists of the following steps:

1. At first, a user creates a `PostgreSQL` database where the data will be restored or the user can use the same `PostgreSQL` database.

2. Then, she creates a `RestoreSession` custom resource that specifies the target `PostgreSQL` database where the backed-up data will be restored. the `Repository` object that points to a `BackupStorage` that holds backend information, and the target `Snapshot`, which will be restored. It also specifies the `Addon` info with task to use to restore.
2. Then, she creates a `RestoreSession` custom resource that specifies the target database where the backed-up data will be restored, addon information (including restore tasks), the target snapshot to be restored, the Repository containing that snapshot, and other additional settings.

3. KubeStash operator watches for `RestoreSession` custom resources.

4. When it finds a `RestoreSession` custom resource, it resolves the respective `Addon` and `Function` and prepares a restore `Job` definition.

5. Then, it creates the `Job` to restore the target.

6. The `Job` reads necessary information to connect with the database from respective `AppBinding` crd. It also reads backend information and access credentials from `Repository` crd and storage `Secret` respectively.
6. The `Job` reads necessary information to connect with the database from respective `AppBinding` CR. It also reads backend information and access credentials from `Repository` CR and storage `Secret` respectively.

7. Then, the `Job` downloads the backed up data from the backend and injects into the desired database. KubeStash pipes the downloaded data to the respective database tool to inject into the database. Hence, restore `Job` does not require a large volume to download entire backup data inside it.

Expand Down

0 comments on commit bd72a71

Please sign in to comment.