Grep-able YAML & JSON

[pbnj]

Problem

I frequently need to grok and grep through YAML and JSON documents.

If you've ever tried grepping through YAML or JSON, you know it's not pleasant due to lost context.

Here is an example grepping through a Helm Chart's values.yaml

$ grep image values.yaml

  image:
    # -- Overrides the Docker registry globally for all images
# -- Image pull secrets for Docker images
imagePullSecrets: []
  image:
    # -- Docker image repository
    # -- Overrides the image tag whose default is the chart's appVersion
    # -- Docker image pull policy
  imagePullSecrets: []
  image:
    # -- The Docker registry for the ingester image. Overrides `loki.image.registry`
    # -- Docker image repository for the ingester image. Overrides `loki.image.repository`
    # -- Docker image tag for the ingester image. Overrides `loki.image.tag`
  image:
    # -- The Docker registry for the distributor image. Overrides `loki.image.registry`
    # -- Docker image repository for the distributor image. Overrides `loki.image.repository`
    # -- Docker image tag for the distributor image. Overrides `loki.image.tag`
  image:
    # -- The Docker registry for the querier image. Overrides `loki.image.registry`
    # -- Docker image repository for the querier image. Overrides `loki.image.repository`
    # -- Docker image tag for the querier image. Overrides `loki.image.tag`
  image:
    # -- The Docker registry for the query-frontend image. Overrides `loki.image.registry`
    # -- Docker image repository for the query-frontend image. Overrides `loki.image.repository`
    # -- Docker image tag for the query-frontend image. Overrides `loki.image.tag`
  image:
    # -- The Docker registry for the table-manager image. Overrides `loki.image.registry`
    # -- Docker image repository for the table-manager image. Overrides `loki.image.repository`
    # -- Docker image tag for the table-manager image. Overrides `loki.image.tag`
  image:
    # -- The Docker registry for the gateway image
    # -- The gateway image repository
    # -- The gateway image tag
    # -- The gateway image pull policy
  image:
    # -- The Docker registry for the compactor image. Overrides `loki.image.registry`
    # -- Docker image repository for the compactor image. Overrides `loki.image.repository`
    # -- Docker image tag for the compactor image. Overrides `loki.image.tag`
    imagePullSecrets: []
  image:
    # -- The Docker registry for the ruler image. Overrides `loki.image.registry`
    # -- Docker image repository for the ruler image. Overrides `loki.image.repository`
    # -- Docker image tag for the ruler image. Overrides `loki.image.tag`
  image:
    # -- Memcached Docker image repository
    # -- Memcached Docker image tag
    # -- Memcached Docker image pull policy
  image:
    # -- Memcached Exporter Docker image repository
    # -- Memcached Exporter Docker image tag
    # -- Memcached Exporter Docker image pull policy

Solution

-o props | --output-format=props to the rescue:

$ yq eval --output-format=props values.yaml | grep image

global.image.registry = null
loki.image.registry = docker.io
loki.image.repository = grafana/loki
loki.image.tag = null
loki.image.pullPolicy = IfNotPresent
ingester.image.registry = null
ingester.image.repository = null
ingester.image.tag = null
distributor.image.registry = null
distributor.image.repository = null
distributor.image.tag = null
querier.image.registry = null
querier.image.repository = null
querier.image.tag = null
queryFrontend.image.registry = null
queryFrontend.image.repository = null
queryFrontend.image.tag = null
tableManager.image.registry = null
tableManager.image.repository = null
tableManager.image.tag = null
gateway.image.registry = docker.io
gateway.image.repository = nginxinc/nginx-unprivileged
gateway.image.tag = 1.19-alpine
gateway.image.pullPolicy = IfNotPresent
compactor.image.registry = null
compactor.image.repository = null
compactor.image.tag = null
ruler.image.registry = null
ruler.image.repository = null
ruler.image.tag = null
memcached.image.registry = docker.io
memcached.image.repository = memcached
memcached.image.tag = 1.6.7-alpine
memcached.image.pullPolicy = IfNotPresent
memcachedExporter.image.registry = docker.io
memcachedExporter.image.repository = prom/memcached-exporter
memcachedExporter.image.tag = v0.6.0
memcachedExporter.image.pullPolicy = IfNotPresent

And yq works with JSON as well:

$ curl -fsSL https://ip-ranges.amazonaws.com/ip-ranges.json | yq eval -o props

syncToken = 1630974796
createDate = 2021-09-07-00-33-16
prefixes.0.ip_prefix = 3.5.140.0/22
prefixes.0.region = ap-northeast-2
prefixes.0.service = AMAZON
prefixes.0.network_border_group = ap-northeast-2
prefixes.1.ip_prefix = 13.34.37.64/27
prefixes.1.region = ap-southeast-4
prefixes.1.service = AMAZON
prefixes.1.network_border_group = ap-southeast-4
...

That's it.

Hope folks find this tip useful as I just happened to discover this flag by accident after searching for a way to tame YAML/JSON documents into a grep-friendly fromat without having to install yet another JSON/YAML cli utility.

Thank you @mikefarah for yq!

[kpym]

@pbnj Do you know gron ?

[pbnj]

Yes, I know gron. In fact, I used to use gron for querying & grepping through YAML files, like: yq -o json | gron.

But I wanted to share that yq -o props can provide that basic use-case without requiring another tool.

[kpym]

@pbnj Indeed -o props is interesting ! There is also yj that can be used like cat in.yaml | yj | gron but the input can be also toml and hcl. And my two cents is to use rg in place of grep ;)