A breaking change update here. Fixes GL #5, #6, and GH #6. The config…

… file format/schema has been updated, and so deployed config files will need to be updated, as per the README.

README.md

@@ -10,17 +10,25 @@ Different phases of the `vzdump` backup can be hooked into, and things can be ru
 The app also logs script output in realtime -- useful when using a long-running process (like `rclone` for example),
 and you want to see progressive timestamping against its output.
 
+>⚠️ **NOTE!** Version 2.0.0 introduces a breaking change to the config file format!  See below.
+
 ## Table of contents
 
 <!-- TOC -->
 * [Proxmox-grapple](#proxmox-grapple)
   * [Table of contents](#table-of-contents)
   * [Purpose and uses](#purpose-and-uses)
+    * [Running binaries](#running-binaries)
+    * [Running things via a shell](#running-things-via-a-shell)
   * [Installation](#installation)
   * [Configuration](#configuration)
     * [Overview](#overview)
     * [Location](#location)
+    * [Configuration dump](#configuration-dump)
     * [Configuration environments](#configuration-environments)
+    * [Breaking change in 2.0.0](#breaking-change-in-200)
+      * [Before 2.0.0 format](#before-200-format)
+      * [New format](#new-format)
   * [Supported versions](#supported-versions)
 <!-- TOC -->
 
@@ -38,31 +46,39 @@ or other apps, to receive status notifications:
 ```yaml
 production:
   job-start:
-    script:
+    mode: script
+    run:
       - "curl -fsS -m 10 --retry 5 -o /dev/null https://your.healthchecks.server/ping/xxx/vzdump-backups/start"
   job-abort:
-    script:
+    mode: script
+    run:
       - "curl -fsS -m 10 --retry 5 -o /dev/null https://your.healthchecks.server/ping/xxx/vzdump-backups/fail"
   backup-abort:
-    script:
+    mode: script
+    run:
       - "curl -fsS -m 10 --retry 5 -o /dev/null https://your.healthchecks.server/ping/xxx/vzdump-backups/fail"
 ```
 
 Maybe you'd like to offsite-sync your backups on job completion:
 
 ```yaml
   job-end:
-    script:
+    mode: script
+    run:
       - "ssh some.host rclone sync --checkers 32 --transfers 16 --dscp cs1 --stats-log-level NOTICE --stats-unit=bits --stats=2m /mnt/pbs-backups remote.host:pbs-rsync"
       - "curl -fsS -m 10 --retry 5 -o /dev/null https://your.healthchecks.server/ping/xxx/vzdump-backups"
 ```
 
+Or maybe as [@lloydbayley](https://github.com/lloydbayley) uses it, via `curl`:
+
+> It actually changes the RGB lights in my rack when a backup happens so it's purely decorative but I like to automate things.
+
 Anything that can be run on the CLI, you can use here.
 
 ### Running things via a shell
 
-Instead of using `script`, you can use `shell`, and anything configured will be run through a shell.  This is directly
-equivalent to the `shell` argument to Python's [`subprocess.Popen`](https://docs.python.org/3/library/subprocess.html#subprocess.Popen).
+Instead of using `mode: script`, you can use `mode: shell`, and anything configured will be run through a shell. This is
+directly equivalent to the `shell` argument to Python's [`subprocess.Popen`](https://docs.python.org/3/library/subprocess.html#subprocess.Popen).
 
 What's the benefit here?  To quote the Python documentation:
 
@@ -96,6 +112,7 @@ the `script` setting:
 script: /home/username/.local/bin/proxmox-grapple
 ```
 
+No other changes to the file is needed.
 
 ## Configuration
 
@@ -106,24 +123,50 @@ script: /home/username/.local/bin/proxmox-grapple
 ```yaml
 production:
   job-end:
-    script:
+    mode: script
+    run:
       - echo 'hi'
       - sleep 1
       - echo 'there'
       - echo 'This is a test.'
 
   backup-end:
+    mode: script
+    run:
     extract:
       enabled: false
       source_directory: /tmp
       destination_directory: /tmp
   #    exclude_storeids:
 
   job-abort:
-    shell:
+    mode: shell
+    run:
       - echo 'your strange command' | tee some_logfile.txt
 ```
 
+Each top-level key should match the different `vzdump` phases.  The currently recognised phases are:
+
+* `job-init`
+* `job-start`
+* `job-end`
+* `job-abort`
+* `backup-start`
+* `backup-end`
+* `backup-abort`
+* `log-end`
+* `pre-stop`
+* `pre-restart`
+* `post-restart`
+
+All of these are optional, and if not configured, are ignored.
+
+If they *are* configured, the accompanying sub-keys are required:
+
+* `mode`
+* `run`
+
+>⚠️ The extract argument is currently not tested, and should be treated as a proof-of-concept only.
 ### Location
 
 The default location for the configuration is `/etc/proxmox_grapple.yml`, or `proxmox_grapple.yml` in the current
@@ -132,6 +175,10 @@ working directory, but this can also be specified on the commandline.
 If a non-absolute path is given, Dynaconf will iterate upwards: it will look at each parent up to the root of the
 system. For each visited folder, it will also try looking inside a `/config` folder.
 
+### Configuration dump
+
+There is a mode, `--dump-config`, that reads all possible configuration and then prints it out for validation purposes.
+
 ### Configuration environments
 
 It can also use different configuration settings based on arbitrary environment names (eg. `production`, `lab`, etc.) It
@@ -145,22 +192,28 @@ For example, to choose a different configuration environment, set the environmen
 ```shell
 root@proxmox:~# ENV_FOR_DYNACONF=lab proxmox-grapple --dump-config
 ```
-Each top-level key should match the different `vzdump` phases.  The currently recognised phases are:
 
-* job-init
-* job-start
-* job-end
-* job-abort
-* backup-start
-* backup-end
-* backup-abort
-* log-end
-* pre-stop
-* pre-restart
-* post-restart
+### Breaking change in 2.0.0
 
->⚠️ The extract argument is currently not tested, and should be treated as a proof-of-concept only.
+To enable the use of the two run modes, I've decided to change the schema of the config file that needs to be updated.
+
+#### Before 2.0.0 format
+
+```yaml
+production:
+  backup-start:
+    script:
+      - echo hi
+```
 
+#### New format
+```yaml
+production:
+  backup-start:
+    mode: script
+    run:
+      - echo hi
+```
 
 ## Supported versions
 

proxmox_grapple_example.yml

@@ -1,12 +1,15 @@
 production:
   job-end:
-    script:
+    mode: script
+    run:
       - echo 'hi'
       - sleep 1
       - echo 'there'
       - echo 'This is a test.'
 
   backup-end:
+    mode: script
+    run:
     extract:
       enabled: false
       source_directory: /tmp

src/proxmox_grapple/config.py

@@ -8,8 +8,54 @@
     core_loaders=['YAML'],
     validators=[
         Validator('job-init', 'job-start', 'job-end', 'job-abort', 'backup-start', 'backup-end', 'backup-abort',
-                  'log-end', 'pre-stop', 'pre-restart', 'post-restart', default={'script': None}
+                  'log-end', 'pre-stop', 'pre-restart', 'post-restart',
+                  is_type_of=dict),
+
+        Validator('job-init.mode', 'job-init.run',
+                  must_exist=True,
+                  when=Validator('job-init', must_exist=True)
+                  ),
+        Validator('job-start.mode', 'job-start.run',
+                  must_exist=True,
+                  when=Validator('job-start', must_exist=True)
+                  ),
+        Validator('job-end.mode', 'job-end.run',
+                  must_exist=True,
+                  when=Validator('job-end', must_exist=True)
+                  ),
+        Validator('job-abort.mode', 'job-abort.run',
+                  must_exist=True,
+                  when=Validator('job-abort', must_exist=True)
+                  ),
+        Validator('backup-start.mode', 'backup-start.run',
+                  must_exist=True,
+                  when=Validator('backup-start', must_exist=True)
+                  ),
+        Validator('backup-end.mode', 'backup-end.run',
+                  must_exist=True,
+                  when=Validator('backup-end', must_exist=True)
                   ),
+        Validator('backup-abort.mode', 'backup-abort.run',
+                  must_exist=True,
+                  when=Validator('backup-abort', must_exist=True)
+                  ),
+        Validator('log-end.mode', 'log-end.run',
+                  must_exist=True,
+                  when=Validator('log-end', must_exist=True)
+                  ),
+        Validator('pre-stop.mode', 'pre-stop.run',
+                  must_exist=True,
+                  when=Validator('pre-stop', must_exist=True)
+                  ),
+        Validator('pre-restart.mode', 'pre-restart.run',
+                  must_exist=True,
+                  when=Validator('pre-restart', must_exist=True)
+                  ),
+        Validator('post-restart.mode', 'post-restart.run',
+                  must_exist=True,
+                  when=Validator('post-restart', must_exist=True)
+                  ),
+
         Validator(
             'backup-end.extract.enabled',
             default=False

src/proxmox_grapple/vzdump_hook_script.py

@@ -20,7 +20,7 @@
 import textwrap
 
 import click
-from dynaconf import ValidationError
+from dynaconf import ValidationError, inspect_settings
 
 from proxmox_grapple.vma_extractor import main as extractor
 from .config import settings
@@ -40,15 +40,16 @@ def dump_config(ctx, param, value):
 
 
 @click.command()
-@click.option('--dump-config', is_flag=True, callback=dump_config, expose_value=False, is_eager=True,
-              help='Dump config as determined by dynaconf')
+# @click.option('--dump-config', is_flag=True, callback=dump_config, expose_value=False, is_eager=True,
+#               help='Dump config as determined by dynaconf')
+@click.option('--dump-config', is_flag=True, help='Dump config after being parsed by Dynaconf')
 @click.option('--config', '-c', required=False, help='Config file path')
 @click.version_option(version=__version__)
 @click.argument('phase')
 @click.argument('mode', required=False)
 @click.argument('vmid', required=False)
 @click.pass_context
-def main(ctx, config, phase, mode, vmid):
+def main(ctx, dump_config, config, phase, mode, vmid):
     """The grappling hook for Proxmox backups.  A more configurable replacement for the Perl version supplied by
     Proxmox Server Solutions GmbH.
 
@@ -74,6 +75,10 @@ def main(ctx, config, phase, mode, vmid):
 
     click.echo(f'HOOK: {" ".join(sys.argv)}')
 
+    if dump_config:
+        click.echo(inspect_settings(settings, print_report=True, dumper="yaml"))
+        sys.exit(0)
+
     # if os.environ.get('STOREID', None) == 'pbs':
     #    click.echo('Detected running in Proxmox Backup Server, exiting')
     #    sys.exit(0)
@@ -102,17 +107,13 @@ def main(ctx, config, phase, mode, vmid):
                 # Extraction not enabled
                 pass
 
-            if any(k in settings[phase] for k in ['script', 'shell']):
-                mode = list(settings[phase].keys())[0]
-                if mode == 'shell':
-                    shell = True
-                else:
-                    shell = False
+            if settings[phase].mode in ['script', 'shell']:
+                shell = settings[phase].mode == 'shell'
 
-                for exec_line in settings[phase][mode]:
+                for exec_line in settings[phase]['run']:
                     try:
-                        click.echo(f'    Running (mode {mode}): {exec_line}')
-                        if mode == 'script':
+                        click.echo(f'    Running (mode {settings[phase].mode}): {exec_line}')
+                        if settings[phase].mode == 'script':
                             exec_line = shlex.split(exec_line)
                         with Popen(exec_line, stdout=PIPE, stderr=STDOUT, text=True, encoding='utf-8', shell=shell) as proc:
                             for line in proc.stdout:
@@ -126,7 +127,7 @@ def main(ctx, config, phase, mode, vmid):
                         click.echo(f'    {e}')
 
         else:
-            click.echo(f'HOOK: Got unknown phase "{phase}", ignoring')
+            click.echo(f'HOOK: Got unknown or unconfigured phase "{phase}", ignoring')
 
     except KeyError as e:
         click.echo(e, err=True)

tests/proxmox_grapple_tests.yml

@@ -1,34 +1,42 @@
 testing:
   job-end:
-    script:
+    mode: script
+    run:
       - echo 'hi'
       - sleep 1
       - echo 'there'
       - echo 'This is a test.'
 
   backup-end:
+    mode: script
+    run:
     extract:
       enabled: true
       source_directory: /tmp
       destination_directory: /tmp
   #    exclude_storeids:
 
   job-start:
-    script:
+    mode: script
+    run:
       - some_missing_command
 
   backup-start:
-    script:
+    mode: script
+    run:
       - /usr/bin/false
 
   post-restart:
-    script:
+    mode: script
+    run:
       derp
 
   backup-abort:
-    shell:
+    mode: shell
+    run:
       - echo 'This is a test' | tr 'i' 'z'
 
   pre-restart:
-    script:
+    mode: script
+    run:
       - echo 'This is a test' | tr 'i' 'z'