Use yq to merge duplicate map keys in a single file

[maartenSXM]

Is there way to use yq to merge duplicate map keys in a single (incorrect) yaml file? yq handles this nicely for multiple yaml files already. For example, if file1.yaml contains this:

splat:
  - blort: glorp
  - foo: bar

and file2.yaml contains this:

splat:
  - blort2: glorp2
  - foo2: bar2

then yq can merge them with yq eval-all '. as $item ireduce ({}; . *+ $item)' file1.yaml file2.yaml to get this:

splat:
  - blort: glorp
  - foo: bar
  - blort2: glorp2
  - foo2: bar2

Although map keys must be unique within a single file, it can be useful if yq could handle duplicates using its merge feature, just as if the duplicate keys were in separate files as above. For example, consider file3.yaml created by cat file1.yaml file2.yaml > file3.yaml so it thus contains:

splat:
  - blort: glorp
  - foo: bar
splat:
  - blort2: glorp2
  - foo2: bar2

yq eval-all '. as $item ireduce ({}; . *+ $item)' file3.yaml yields:

splat:
  - blort2: glorp2
  - foo2: bar2

Passing file3.yaml twice using, yq eval-all '. as $item ireduce ({}; . *+ $item)' file3.yaml file3.yaml yields:

splat:
  - blort2: glorp2
  - foo2: bar2
  - blort2: glorp2
  - foo2: bar2

However, the desired result is the same as when the duplicate map keys are in separate files:

splat:
  - blort: glorp
  - foo: bar
  - blort2: glorp2
  - foo2: bar2

Is there an incantation to yq that will take a single (incorrect) yaml file such as file3.yaml above and merge the duplicate map keys?

Thank you.

[maartenSXM]

A workaround to do this in two steps is to split the yaml file with duplicate map keys into multiple files each with a single map key and then run yq to merge them. Like this:

awk '/^[^ ]/{close(F);F=FILENAME"."NR;print $0>F};/^ /{print $0>F}' file3.yaml
yq eval-all '. as $item ireduce ({}; . *+ $item)' file3.yaml.*

However, I suspect this is possible with yq in one step, somehow. Anyone know the yq grammar well enough to say?

[mikefarah]

Sorry - because it's (as you pointed out) not valid yaml; the underlying yaml decoder doesn't support reading both those values for the same map key. You'd have to do a pre-processing step like @maartenSXM has got and produce valid yaml content before using yq.

Note that you could also split the file into several 'documents' like so:

splat:
  - blort: glorp
  - foo: bar
---
splat:
  - blort2: glorp2
  - foo2: bar2

And then merge those documents together...

[maartenSXM]

Hi @mikefarah, Your reply helped me reduce the number of temporary files created down to one by doing this:

awk '/^[^[:space:]]/{print "---"};{print $0}' file3.yaml > file3.yaml.tmp && mv file3.yaml.tmp file3.yaml
yq --inplace eval-all '. as $item ireduce ({}; . *+ $item)' file3.yaml

This solution will scale to very large numbers of map keys since it does not generate one file per map key. Thank you.