New fields added to .value in to_entries[] | (.value changes) | from_entries are lost if I combine the result with another array

[Frederick888]

I'm trying to add matrices into jobs in a GitHub Actions manifest, but only to those which have actions/setup-go@... steps. (I'm using yq v4.44.1 from Homebrew.)

The input looks like:

name: CI
on:
  pull_request:
jobs:
  no_setup_go:
    runs-on: [ubuntu-stable]
    steps:
      - run: |
          exit 0
  has_setup_go:
    runs-on: [ubuntu-stable]
    steps:
      - uses: actions/setup-go@v3
        with:
          go-version: "1.20.1"
      - run: |
          exit 0
  also_has_setup_go:
    runs-on: [ubuntu-stable]
    strategy:
      fail-fast: false
      matrix:
        go_version:
          - "1.19"
          - "1.20"
          - "1.21"
    steps:
      - uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go_version }}
      - run: |
          exit 0

Since

GO_VERSIONS='["1.20.x", "1.21.x", "1.22.x"]'

yq --exit-status eval '
    .jobs
        | to_entries[]
        | select(.value.steps[] | select(.uses | contains("/setup-go@")) | . += { "with": { "go-version": "${{ matrix.go_version }}" } })
        | .value.strategy.fail-fast = false | .value.strategy.matrix.go_version = '"$GO_VERSIONS"'
    ' .github/workflows/ci.yaml

...gave me

key: has_setup_go
value:
  runs-on: [ubuntu-stable]
  steps:
    - uses: actions/setup-go@v3
      with:
        go-version: ${{ matrix.go_version }}
    - run: |
        exit 0
  strategy:
    fail-fast: false
    matrix:
      go_version:
        - 1.20.x
        - 1.21.x
        - 1.22.x
key: also_has_setup_go
value:
  runs-on: [ubuntu-stable]
  strategy:
    fail-fast: false
    matrix:
      go_version:
        - 1.20.x
        - 1.21.x
        - 1.22.x
  steps:
    - uses: actions/setup-go@v3
      with:
        go-version: ${{ matrix.go_version }}
    - run: |
        exit 0

I thought it'd be a simple matter of merging back the other jobs:

GO_VERSIONS='["1.20.x", "1.21.x", "1.22.x"]'

yq --exit-status eval '
    .jobs = (
        [
            .jobs
                | to_entries[]
                | select(.value.steps[] | select(.uses | contains("/setup-go@")) | . += { "with": { "go-version": "${{ matrix.go_version }}" } })
                | .value.strategy.fail-fast = false | .value.strategy.matrix.go_version = '"$GO_VERSIONS"'
        ] + [
            .jobs
                | to_entries[]
                | select([.value.steps[] | select(.uses | contains("/setup-go@"))] | length == 0)
        ]
        | from_entries
    )' .github/workflows/ci.yaml

However this actually gave me:

name: CI
on:
  pull_request:
jobs:
  has_setup_go:
    runs-on: [ubuntu-stable]
    steps:
      - uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go_version }}
      - run: |
          exit 0
  also_has_setup_go:
    runs-on: [ubuntu-stable]
    strategy:
      fail-fast: false
      matrix:
        go_version:
          - 1.20.x
          - 1.21.x
          - 1.22.x
    steps:
      - uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go_version }}
      - run: |
          exit 0
  no_setup_go:
    runs-on: [ubuntu-stable]
    steps:
      - run: |
          exit 0

The new strategy field in has_setup_go was somehow gone. In fact, I could replace the second array of jobs with an empty [] and the same thing happened.

Any idea what I did wrong? Thank you.

[Frederick888]

The workaround I had later was

GO_VERSIONS='["1.20.x", "1.21.x", "1.22.x"]'

# shellcheck disable=SC2016
# shellcheck disable=SC2016
yq --exit-status '
    .jobs = (
        [
            .jobs
                | to_entries[]
                | select(.value.steps[] | select(.uses | contains("/setup-go@")) | . += { "with": { "go-version": "${{ matrix.go_version }}" } })
                | .value = { "strategy": { "fail-fast": false, "matrix": { "go_version": '"$GO_VERSIONS"' } } } + (.value | with_entries(select(.key != "strategy")))  # Add strategy to the top
        ] + [
            .jobs
                | to_entries[]
                | select([.value.steps[] | select(.uses | contains("/setup-go@"))] | length == 0)
        ]
        | from_entries
    )' -i .github/workflows/ci.yaml

It unfortunately doesn't preserve the original key order (I'd like to avoid unnecessary changes as I want to Git-commit them.)

[mikefarah]

You can do it like this:

yq 'with(.jobs[] | select(.steps[].uses | contains("/setup-go")); 
     .with.go-version="mike" | .strategy.fail-fast=false
)'  examples/data1.yaml

Explanation:

• Use with to make multiple updates to a deep path https://mikefarah.gitbook.io/yq/operators/with
• Splat jobs to match all the values, then you can filter those value

*formatting

[Frederick888]

Thank you! FWIW, my final script was

yq --exit-status eval '
    with(
        .jobs[] | select(.steps[] | select(.uses | contains("/setup-go@")) | .with.go-version = "${{ matrix.go_version }}");
        .strategy.fail-fast = false
            | .strategy.matrix.go_version = '"$GO_VERSIONS"'
    )' .github/workflows/ci.yaml

May I know why my original solution was broken? And e.g.

yq --exit-status eval '
    .jobs = [with(
        .jobs[] | select(.steps[] | select(.uses | contains("/setup-go@")) | .with.go-version = "${{ matrix.go_version }}");
        .strategy.fail-fast = false
            | .strategy.matrix.go_version = '"$GO_VERSIONS"'
    )]' .github/workflows/ci.yaml

...also didn't work.