Merge pull request #5 from grahampugh/autopkg_formatting

Autopkg formatting

CHANGELOG.md

@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file. This projec
 
 ## [unreleased]
 
+Changes since last release will be listed here.
+
+## [v0.6.0] - 2021-02-23 - v0.6.0
+
+- When converting an AutoPkg recipe to `yaml` format, specific formatting is carried out:
+  - The different process dictionaries are ordered by Processor, Comment, Arguments.
+  - The Input dictionary is ordered such that NAME is always at the top.
+  - The items are ordered thus: Comment, Description, Identifier, ParentRecipe, MinimumVersion, Input, Process
+  - Blank lines are added for human readability. Specifically these are added above Input and Process dictionaries, and between each Processor dictionary.
+- You can use `yaml_tidy.py` to reformat existing `.recipe.yaml` files as above.
+- An entire directory structure can have `.recipe.yaml` files reformatted as above using the command `plistyamlplist /path/to/YAML --tidy`. Any directory with a `YAML` or `_YAML` folder in it will be processed, including subdirectories.
+
 ## [v0.5.0] - 2021-02-11 - v0.5.0
 
 - Switched from `pyyaml` to `ruamel.yaml`.
@@ -36,7 +48,8 @@ All notable changes to this project will be documented in this file. This projec
 
 - Initial Release (though the tool has been around for some time).
 
-[unreleased]: https://github.com/grahampugh/plist-yaml-plist/compare/v0.5.0...HEAD
+[unreleased]: https://github.com/grahampugh/plist-yaml-plist/compare/v0.6.0...HEAD
+[v0.6.0]: https://github.com/grahampugh/plist-yaml-plist/compare/v0.5.0...v0.6.0
 [v0.5.0]: https://github.com/grahampugh/plist-yaml-plist/compare/v0.4.0...v0.5.0
 [v0.4.0]: https://github.com/grahampugh/plist-yaml-plist/compare/v0.3.1...v0.4.0
 [v0.3.1]: https://github.com/grahampugh/plist-yaml-plist/compare/v0.3...v0.3.1

README.md

@@ -24,7 +24,7 @@ plistyamlplist -h
 Usage: ./plistyamlplist.py <input-file> [<output-file>]
 ```
 
-You can supply the input-file as a glob (`*.yaml` or `*.json`) to convert an entire directory or subset of `yaml` or `json` files. This currently only work for converting from yaml to plist. Note that you have to escape the glob, i.e. write as `plistyamlplist /path/to/\*.yaml`.
+You can supply the input-file as a glob (`*.yaml` or `*.json`) to convert an entire directory or subset of `yaml` or `json` files. This currently only work for converting from yaml to plist. Note that you have to escape the glob, i.e. write as `plistyamlplist /path/to/\*.yaml`. Or, just supply a folder. The folder must be `_YAML` or `YAML` or a subfolder of one of these.
 Otherwise, each file can be used individually:
 
 ```bash
@@ -85,6 +85,38 @@ If the above folder does not exist, you will be prompted to create it.
 
 If there is no `YAML`/`JSON` folder in the path, the converted file will be placed in the same folder.
 
+## Special handling of AutoPkg recipes
+
+If you convert an AutoPkg recipe from `plist` to `yaml`, the following formatting is carried out:
+
+- The different process dictionaries are ordered by Processor, Comment, Arguments (python3 only).
+- The Input dictionary is ordered such that NAME is always at the top (python3 only).
+- The items are ordered thus: Comment, Description, Identifier, ParentRecipe, MinimumVersion, Input, Process (python3 only).
+- Blank lines are added for human readability. Specifically these are added above Input and Process dictionaries, and between each Processor dictionary.
+
+You can also carry out reformatting of existing `yaml` recipes using the `yaml_tidy.py` script, or using `plistyamlplist` as in the following examples:
+
+- Convert an AutoPkg recipe to yaml format:
+
+  plistyamlplist /path/to/SomeRecipe.recipe
+
+- Reformat a single `yaml`-based recipe:
+
+  plistyamlplist /path/to/SomeRecipe.recipe.yaml --tidy
+
+- Reformat a an entire folder structure containing `yaml`-based recipes:
+
+  ```bash
+  plistyamlplist /path/to/YAML/ --tidy
+  # this will process all .recipe.yaml files in the folders within /path/to/YAML
+
+  plistyamlplist /path/to/\_YAML/ --tidy
+  # this will process all .recipe.yaml files in the folders within /path/to/_YAML
+
+  plistyamlplist /path/to/YAML/subfolder/ --tidy
+  # this will process all .recipe.yaml files in the folders within /path/to/_YAML/subfolder
+  ```
+
 ## Credits
 
 Elements of these scripts come from:

pkg/plistyamlplist/build-info.plist

@@ -17,6 +17,6 @@
 	<key>suppress_bundle_relocation</key>
 	<true/>
 	<key>version</key>
-	<string>0.5.0</string>
+	<string>0.6.0</string>
 </dict>
 </plist>

plistyamlplist.py

@@ -13,40 +13,60 @@
 
 import sys
 import os
+import shutil
 import glob
+import re
 
 
 from plistyamlplist_lib.plist_yaml import plist_yaml
 from plistyamlplist_lib.yaml_plist import yaml_plist
 from plistyamlplist_lib.json_plist import json_plist
+from plistyamlplist_lib.yaml_tidy import tidy_yaml
 
 
 def usage():
     """print help."""
-    print("Usage: plistyamlplist.py <input-file> <output-file>\n")
+    print("Usage: plistyamlplist.py <input> <output>\n")
     print(
-        "If <input-file> is a PLIST and <output-file> is omitted,\n"
-        "<input-file> is converted to <input-file>.yaml\n"
+        "If <input> is a PLIST file and <output> is omitted,\n"
+        "<input> is converted to <input>.yaml in the same folder.\n"
     )
     print(
-        "If <input-file> ends in .yaml or .yml and <output-file> is omitted,\n"
-        "<input-file>.yaml is converted to PLIST format with name <input-file>\n"
+        "If <input> ends in .yaml or .yml and <output> is omitted,\n"
+        "<input>.yaml is converted to PLIST format with name <input>\n"
     )
+    print(
+        "If <input> is a folder with 'YAML' or 'JSON' in the path,\n"
+        "all yaml files in the subfolders will be converted to plists in\n"
+        "the corresponding subfolder structure above the YAML or JSON file.\n"
+        "Or, if <output> is specified as another folder, the corresponding\n"
+        "folder structure will be reproduced under the <output> folder"
+    )
+    print(
+        "If <input> is a folder with 'PLIST' in the path,\n"
+        "and <output> is specified as another folder,"
+        "all yaml files in the subfolders will be converted to plist in\n"
+        "the corresponding subfolder structure under the <output> folder."
+    )
+    print("If <output> is --tidy,\n" "<input>.yaml is tidied up for AutoPkg.\n")
 
 
 def check_if_plist(in_path):
     """rather than restrict by filename, check if the file is a plist by
     reading the second line of the file for the PLIST declaration."""
     with open(in_path) as fp:
-        for i, line in enumerate(fp):
-            if i == 1:
-                # print line
-                if line.find("PLIST 1.0") == -1:
-                    is_plist = False
-                else:
-                    is_plist = True
-            elif i > 2:
-                break
+        try:
+            for i, line in enumerate(fp):
+                if i == 1:
+                    # print line
+                    if line.find("PLIST 1.0") == -1:
+                        is_plist = False
+                    else:
+                        is_plist = True
+                elif i > 2:
+                    break
+        except UnicodeDecodeError:
+            is_plist = False
     return is_plist
 
 
@@ -112,7 +132,7 @@ def get_out_path(in_path, filetype):
             out_path = filename
     else:
         if check_if_plist(in_path):
-            out_path = "{}.yaml".format(in_path)
+            out_path = in_path + ".yaml"
         else:
             print("\nERROR: File is not PLIST, JSON or YAML format.\n")
             usage()
@@ -160,7 +180,10 @@ def main():
                 out_path = sys.argv[2]
             if filetype == "yaml":
                 print("Processing yaml file...")
-                yaml_plist(in_path, out_path)
+                if out_path == "--tidy":
+                    tidy_yaml(in_path)
+                else:
+                    yaml_plist(in_path, out_path)
             elif filetype == "json":
                 print("Processing json file...")
                 json_plist(in_path, out_path)
@@ -169,17 +192,104 @@ def main():
     elif os.path.isdir(in_path) and "YAML" in in_path:
         print("Processing YAML folder...")
         filetype = "yaml"
-        for in_file in os.listdir(in_path):
-            in_file_path = os.path.join(in_path, in_file)
-            out_path = get_out_path(in_file_path, filetype)
-            yaml_plist(in_file_path, out_path)
+        if sys.argv[2] == "--tidy":
+            print("WARNING! Processing all subfolders...\n")
+            for root, dirs, files in os.walk(in_path):
+                for name in files:
+                    tidy_yaml(os.path.join(root, name))
+                for name in dirs:
+                    tidy_yaml(os.path.join(root, name))
+        elif os.path.isdir(sys.argv[2]):
+            # allow batch replication of folder structure and conversion of yaml to plist
+            # also copies other file types without conversion to the same place in the
+            #  hierarchy
+            out_path_base = os.path.abspath(sys.argv[2])
+            print("Writing to {}".format(out_path_base))
+            for root, dirs, files in os.walk(in_path):
+                for name in dirs:
+                    working_dir = os.path.join(out_path_base, name)
+                    if not os.path.isdir(working_dir):
+                        print("Creating new folder " + working_dir)
+                        os.mkdir(working_dir)
+                for name in files:
+                    source_path = os.path.join(root, name)
+                    print("In path: " + in_path)
+                    sub_path = re.sub(in_path, "", source_path)
+                    print("Subdirectory path: " + sub_path)
+                    filename, _ = os.path.splitext(
+                        os.path.join(out_path_base, sub_path)
+                    )
+                    print("Source path: " + source_path)
+                    if source_path.endswith(".yaml"):
+                        dest_path = filename + ".plist"
+                        print("Destination path for plist: " + dest_path)
+                        yaml_plist(source_path, dest_path)
+                    else:
+                        dest_path = os.path.join(os.path.join(out_path_base, sub_path))
+                        print("Destination path: " + dest_path)
+                        try:
+                            shutil.copy(source_path, dest_path)
+                            if os.path.isfile(dest_path):
+                                print("Written to " + dest_path + "\n")
+                        except IOError:
+                            print("ERROR: could not copy " + source_path + "\n")
+        else:
+            for in_file in os.listdir(in_path):
+                in_file_path = os.path.join(in_path, in_file)
+                out_path = get_out_path(in_file_path, filetype)
+                yaml_plist(in_file_path, out_path)
     elif os.path.isdir(in_path) and "JSON" in in_path:
         print("Processing JSON folder...")
         filetype = "json"
         for in_file in os.listdir(in_path):
             in_file_path = os.path.join(in_path, in_file)
             out_path = get_out_path(in_file_path, filetype)
             json_plist(in_file_path, out_path)
+    elif os.path.isdir(in_path) and "PLIST" in in_path:
+        print("Processing PLIST folder...")
+        filetype = "plist"
+        if os.path.isdir(sys.argv[2]):
+            # allow batch replication of folder structure and conversion of plist to yaml
+            # also copies other file types without conversion to the same place in the
+            # hierarchy
+            out_path_base = os.path.abspath(sys.argv[2])
+            print("Writing to " + out_path_base)
+            for root, dirs, files in os.walk(in_path):
+                for name in dirs:
+                    source_dir = os.path.join(root, name)
+                    sub_dir = re.sub(in_path, "", source_dir)
+                    working_dir = out_path_base + sub_dir
+                    if "YAML" in working_dir:
+                        # chances are we don't want to copy the contents of a YAML
+                        # folder here
+                        continue
+                    if not os.path.isdir(working_dir):
+                        print("Creating new folder " + working_dir)
+                        os.mkdir(working_dir)
+                for name in files:
+                    source_path = os.path.join(root, name)
+                    if "YAML" in source_path:
+                        # chances are we don't want to copy the contents of a YAML
+                        # folder here
+                        continue
+                    print("In path: " + in_path)
+                    sub_path = re.sub(in_path, "", source_path)
+                    print("Subdirectory path: " + sub_path)
+                    print("Source path: " + source_path)
+                    if check_if_plist(source_path):
+                        filename = re.sub(".plist", "", out_path_base + sub_path)
+                        dest_path = filename + ".yaml"
+                        print("Destination path for yaml: " + dest_path)
+                        plist_yaml(source_path, dest_path)
+                    else:
+                        dest_path = out_path_base + sub_path
+                        print("Destination path: " + dest_path)
+                        try:
+                            shutil.copy(source_path, dest_path)
+                            if os.path.isfile(dest_path):
+                                print("Written to " + dest_path + "\n")
+                        except IOError:
+                            print("ERROR: could not copy " + source_path + "\n")
     else:
         if check_if_plist(in_path):
             try:

plistyamlplist_lib/handle_autopkg_recipes.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from collections import OrderedDict
+
+
+def optimise_autopkg_recipes(recipe):
+    """If input is an AutoPkg recipe, optimise the yaml output in 3 ways to aid
+    human readability:
+
+    1. Adjust the Processor dictionaries such that the Comment and Arguments keys are
+       moved to the end, ensuring the Processor key is first.
+    2. Ensure the NAME key is the first item in the Input dictionary.
+    3. Order the items such that the Input and Process dictionaries are at the end.
+    """
+
+    if "Process" in recipe:
+        process = recipe["Process"]
+        new_process = []
+        for processor in process:
+            processor = OrderedDict(processor)
+            if "Comment" in processor:
+                processor.move_to_end("Comment")
+            if "Arguments" in processor:
+                processor.move_to_end("Arguments")
+            new_process.append(processor)
+        recipe["Process"] = new_process
+
+    if "Input" in recipe:
+        input = recipe["Input"]
+        if "NAME" in input:
+            input = OrderedDict(reversed(list(input.items())))
+            input.move_to_end("NAME")
+        recipe["Input"] = OrderedDict(reversed(list(input.items())))
+
+    desired_order = [
+        "Comment",
+        "Description",
+        "Identifier",
+        "ParentRecipe",
+        "MinimumVersion",
+        "Input",
+        "Process",
+    ]
+    desired_list = [k for k in desired_order if k in recipe]
+    reordered_recipe = {k: recipe[k] for k in desired_list}
+    reordered_recipe = OrderedDict(reordered_recipe)
+    return reordered_recipe
+
+
+def format_autopkg_recipes(output):
+    """Add lines between Input and Process, and between multiple processes.
+    This aids readability of yaml recipes"""
+    # add line before specific processors
+    for item in ["Input:", "Process:", "- Processor:"]:
+        output = output.replace(item, "\n" + item)
+
+    # remove line before first process
+    output = output.replace("Process:\n\n-", "Process:\n-")
+
+    recipe = []
+    lines = output.splitlines()
+    for line in lines:
+        # convert quoted strings with newlines in them to scalars
+        if "\\n" in line:
+            spaces = len(line) - len(line.lstrip()) + 2
+            print(spaces)
+            space = " "
+            line = line.replace(': "', ": |\n{}".format(space * spaces))
+            line = line.replace("\\t", "    ")
+            line = line.replace('\\n"', "")
+            line = line.replace("\\n", "\n{}".format(space * spaces))
+            line = line.replace('\\"', '"')
+            if line[-1] == '"':
+                line[:-1]
+        # elif "%" in lines:
+        #  handle strings that have AutoPkg %percent% variables in them
+        # (these need to be quoted)
+
+        # print(line)
+        recipe.append(line)
+    recipe.append("")
+    # print("\n".join(recipe))
+    return "\n".join(recipe)