What is the concept behind the different kind of comments?

[towi]

Excellent tool! I like command-line manipulating structured files, always have!

I want to preserve the comments when I apply sort_keys recursivly, because I want to do a diff. Exactly the use-case you suggested in the docs.

But I am confused about the handling of comments before yaml lines. Those are shuffled around without me seeing by what rules.

I noticed there are head_comments, line_comments and foot_comments. But there is also also comments="", so there is another name?

I was guessing head_comments could be the comments before a yaml-line. But from the examples it looks like they are the comments at the top of a document?

This would fit to the foot_comments example which seem to me always at the end of a document.

---

What I would need would be a guideline on how I should design my YAML files, ie. the comments therein, that a .. sort_keys for a diff would not rearrange the comments.

I will give you an example what I tried.

File: values-ex.yaml

#################################################################################
## My Special Values File
#################################################################################

# Docker Image
image:
  tag: 3.14-SNAPSHOT
  timestamp: "20220530T1203"

# Incoming
# - the "subdomain" is needed, because they said so.
incoming:
  enabled: true
  hosts:
    - hostname: "*.example.com"
      patten: letters
    - hostname: "www.example.com"
      pattern: digits
    - hostname: "ftp.example.com"
      pattern: none

# injected into env
env:
  framework:
    component:
      #key: a,b,c
      key: b,c

# injected into header
header:
  # the header identifier
  identifier: "test-id"
  # the header roundtrip
  roundtrip: "19g"
  # stuffing frame:
  stuffingFrameIntro: "U21X"
  stuffingFrameOutro: "X12U"
  # drain
  stuffingDrainUfo: "a=12;b;13//foo"
  stuffingDrainPlace: "ffee-aa08"
  stuffingDrainOrtho: "Humor"
  # the important app id:
  stuffingApplicationId: "111222333"
  # we need the app ref too:
  stuffingApplicationRef: "6"
  # setting the app source:
  stuffingApplicationSource: "aaab-98be"
  # krinskrams overwrite here:
  krimskrams: 'com.example.ClassName needed in="77s" out="AA77 18dx";'

has the output of yq 'sort_keys(..)' values-ex.yaml > values-ex.yaml.sorted

File: values-ex.yaml.sorted

#################################################################################
## My Special Values File
#################################################################################

# Docker Image
# injected into env
env:
  framework:
    component:
      #key: a,b,c
      key: b,c
# injected into header
header:
  # the header identifier
  identifier: "test-id"
  # the header roundtrip

  krimskrams: 'com.example.ClassName needed in="77s" out="AA77 18dx";'
  roundtrip: "19g"
  # stuffing frame:

  stuffingApplicationId: "111222333"
  # we need the app ref too:

  stuffingApplicationRef: "6"
  # setting the app source:

  stuffingApplicationSource: "aaab-98be"
  # krinskrams overwrite here:

  stuffingDrainOrtho: "Humor"
  # the important app id:

  stuffingDrainPlace: "ffee-aa08"
  stuffingDrainUfo: "a=12;b;13//foo"
  stuffingFrameIntro: "U21X"
  stuffingFrameOutro: "X12U"
  # drain
image:
  tag: 3.14-SNAPSHOT
  timestamp: "20220530T1203"
# Incoming

# - the "subdomain" is needed, because they said so.
incoming:
  enabled: true
  hosts:
    - hostname: "*.example.com"
      patten: letters
    - hostname: "www.example.com"
      pattern: digits
    - hostname: "ftp.example.com"
      pattern: none

Things to note here:

• # Docker Image comment should have been kept in front of image:
• # Incoming seems to have been kept with image: as some kind of foot_comment? But it should have been kept with incoming:.
• Similar with # the header roundtrip which seems to have been kept as a foot_comment, but should have been kept near roundtrip: (would this be considered a head_comment of roundtrip?)

This would be my roughly expected output:

#################################################################################
## My Special Values File
#################################################################################

# injected into env
env:
  framework:
    component:
      #key: a,b,c
      key: b,c

# injected into header
header:
  # the header identifier
  identifier: "test-id"

  # krinskrams overwrite here:
  krimskrams: 'com.example.ClassName needed in="77s" out="AA77 18dx";'

  # the header roundtrip
  roundtrip: "19g"

  # the important app id:
  stuffingApplicationId: "111222333"
  # we need the app ref too:
  stuffingApplicationRef: "6"
  # setting the app source:
  stuffingApplicationSource: "aaab-98be"

  stuffingDrainOrtho: "Humor"
  stuffingDrainPlace: "ffee-aa08"
  # drain
  stuffingDrainUfo: "a=12;b;13//foo"

  # stuffing frame:
  stuffingFrameIntro: "U21X"
  stuffingFrameOutro: "X12U"

# Docker Image
image:
  tag: 3.14-SNAPSHOT
  timestamp: "20220530T1203"

# Incoming
# - the "subdomain" is needed, because they said so.
incoming:
  enabled: true
  hosts:
    - hostname: "*.example.com"
      patten: letters
    - hostname: "www.example.com"
      pattern: digits
    - hostname: "ftp.example.com"
      pattern: none

The newlines are arbitrary here. (I rearranged the file manually, if there is an error by my manual work you probably know what I meant, anyway.)

Note that I put the comment # drain in front of stuffingDrainUfo:. A human would of course recognize that # drain was the comment for a whole group if lines, but the computer can't know this. So it went with the rearrangement of the lines in the group.

---

If I would understand the concept behind comments I could maybe design my YAML files in a way that the comments are kept at their keyss, even after sorting.

Or maybe the parsing of comments should be redesigned? It seems that often a comment belongs to the line in front of it, which I find a bit unusual?

Or maybe empty lines have meaning for comment parsing?

I would also benefit from an option to output the internal parse-tree (AST, if you have such a thing). Then I could understand what my comments are interpreted as and where they are assigned to. I tried to do that with the -o=xml option but that wasn't detailed enough.

---

Thanks in advance,
towi.

[mikefarah]

Hi - thanks for your feedback! I'm a little limited to my time atm, so I'll have to keep this short - I appreciate your detail though.

headComment / lineComment / footComment are parsed directly from the underlying yaml parser (go-yaml). They should represent exactly that according to their node - however go-yaml just makes a best effort and it's not 100% accurate unfortunately :(

I dont have a built in AST representation visible, however you can get similar data by using an expression like:

yq '... | {"path": path, "headComment": headComment, "lineComment": lineComment, "footComment": footComment}'

I don't have yq available on the terminal I'm on - so hopefully I haven't made a typo.

[mikefarah]

Quick update: the command should be:

yq '[... | {"path": (path | join(".")), "hc": headComment, "fc": footComment, "lc": lineComment, "tag": tag}]' examples/data1.yaml

It's a bit rough - note that you will see two entries for a given path, first the key/index, then the value.