Add command to generate jsonschema for limayaml #2306

afbjorklund · 2024-04-30T09:43:39Z

https://pypi.org/project/check-jsonschema/

Actually found two bugs, with the current code:

Schema validation errors were encountered.
  examples/default.yaml::$.vmType: None is not of type 'string'
  examples/default.yaml::$.os: None is not of type 'string'
  examples/default.yaml::$.arch: None is not of type 'string'
  examples/default.yaml::$.cpuType.armv7l: None is not of type 'string'
  examples/default.yaml::$.cpuType.aarch64: None is not of type 'string'
  examples/default.yaml::$.cpuType.x86_64: None is not of type 'string'
  examples/default.yaml::$.cpus: None is not of type 'integer'
  examples/default.yaml::$.memory: None is not of type 'string'
  examples/default.yaml::$.disk: None is not of type 'string'

Schema validation errors were encountered.
  examples/docker.yaml::$.probes[0]: Additional properties are not allowed ('hint', 'script' were unexpected)
  examples/docker.yaml::$.probes[0]: 'Mode' is a required property
  examples/docker.yaml::$.probes[0]: 'Description' is a required property
  examples/docker.yaml::$.probes[0]: 'Script' is a required property
  examples/docker.yaml::$.probes[0]: 'Hint' is a required property

But the rest of it is just about adding "nullable".

To test this feature: (hidden command: limactl generate-jsonschema)

make schema-limayaml.json

And use it to test: (https://check-jsonschema.readthedocs.io/)

make check-jsonschema

mac2net · 2024-04-30T10:35:35Z

Don't you think this universal rejection of “none” is more an opinionated code base rather than a validation error?

afbjorklund · 2024-04-30T12:35:10Z

Don't you think this universal rejection of “none” is more an opinionated code base rather than a validation error?

It's not a rejection (as such), there is a patch for it (to not have to decorate the go struct) but it has not been merged yet

feat: Add NullableFromType option invopop/jsonschema#110

With the patch, one could (perhaps) mark all string pointers and arrays as accepting null as a valid value (by default).

feat: Add NullableFromJSONSchemaTags option invopop/jsonschema#106

But it does catch some issues, with string being fed with null - which is valid in JavaScript but not really in Go...

Having to decorate the pointers and arrays I think is not so intrusive, so it seems like a valid workaround to me.

                "vmType": {
                    "type": "string"
                },

examples/default.yaml::$.os: None is not of type 'string'

With "nullable", the following json schema is generated instead:

                "vmType": {
                    "oneOf": [
                        {
                            "type": "string"
                        },
                        {
                            "type": "null"
                        }
                    ]
                },

afbjorklund · 2024-05-01T07:12:40Z

$ make check-jsonschema 
# The hostagent must be compiled with CGO_ENABLED=1 so that net.LookupIP() in the DNS server
# calls the native resolver library and not the simplistic version in the Go library.
CGO_ENABLED=1 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v0.21.0-51-ga177cd8" -tags "" -o _output/bin/limactl ./cmd/limactl
_output/bin/limactl generate-schema >schema-limayaml.json
check-jsonschema --schemafile schema-limayaml.json examples/default.yaml
ok -- validation done

These arrays have to allow null, since they are given that way in the default yaml:

additionalDisks:
# disks should either be a list of disk name strings, for example:
# - "data"
# or a list of disk objects with extra parameters, for example:
# - name: "data"
#   format: true
#   fsType: "ext4"

caCerts:
  ...

  # A list of trusted CA certificate files. The files will be read and passed to cloud-init.
  files:
  # - examples/hello.crt

  # A list of trusted CA certificates. These are directly passed to cloud-init.
  certs:
  # - |
  #   -----BEGIN CERTIFICATE-----
  #   YOUR-ORGS-TRUSTED-CA-CERT-HERE
  #   -----END CERTIFICATE-----
  # - |
  #   -----BEGIN CERTIFICATE-----
  #   YOUR-ORGS-TRUSTED-CA-CERT-HERE
  #   -----END CERTIFICATE-----

networks:
# Lima can manage daemons for networks defined in $LIMA_HOME/_config/networks.yaml
# automatically. The socket_vmnet binary must be installed into
# secure locations only alterable by the "root" user.
# The same applies to vde_switch and vde_vmnet for the deprecated VDE mode.
# - lima: shared
#   # MAC address of the instance; lima will pick one based on the instance name,
#   # so DHCP assigned ip addresses should remain constant over instance restarts.
#   macAddress: ""
#   # Interface name, defaults to "lima0", "lima1", etc.
#   interface: ""
#
# Lima can also connect to "unmanaged" networks addressed by "socket". This
# means that the daemons will not be controlled by Lima, but must be started
# before the instance.  The interface type (host, shared, or bridged) is
# configured in socket_vmnet and not in lima.
# - socket: "/var/run/socket_vmnet"

hostResolver:

  ...
  hosts:
    # guest.name: 127.1.1.1
    # host.name: host.lima.internal

A better alternative would be to comment out the key as well, and leave them undefined?

The other keys are only allowing null values, when they explicitly are given that way:

vmType: null
os: null
arch: null
images:
  - location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img
    arch: x86_64
    digest: sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3
  - location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img
    arch: aarch64
    digest: sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22
  - location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
    arch: x86_64
  - location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
    arch: aarch64
cpus: null
memory: null
disk: null
mounts:
  - location: "~"
    mountPoint: null
    writable: null
    sshfs:
      cache: null
      followSymlinks: null
      sftpDriver: null
    9p:
      securityModel: null
      protocolVersion: null
      msize: null
      cache: null
  - location: /tmp/lima
    writable: true
mountType: null
mountInotify: null
additionalDisks: null
ssh:
  localPort: 0
  loadDotSSHPubKeys: null
  forwardAgent: null
  forwardX11: null
  forwardX11Trusted: null
caCerts:
  removeDefaults: null
  files: null
  certs: null
upgradePackages: null
containerd:
  system: null
  user: null
cpuType:
  aarch64: null
  armv7l: null
  x86_64: null
rosetta:
  enabled: null
  binfmt: null
timezone: null
firmware:
  legacyBIOS: null
audio:
  device: null
video:
  display: null
  vnc:
    display: null
networks: null
propagateProxyEnv: null
hostResolver:
  enabled: null
  ipv6: null
  hosts: null
guestInstallPrefix: null
plain: null

Currently the only required key is "images", but even that might be optional (for ext).

afbjorklund · 2024-05-01T18:46:36Z

Here is the empty yaml:

images: []

And here is default yaml:

images:
- location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img
  arch: x86_64
  digest: sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3
- location: https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img
  arch: aarch64
  digest: sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22
- location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
  arch: x86_64
- location: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
  arch: aarch64
mounts:
- location: ~
- location: /tmp/lima
  writable: true

Hopefully the images can be made more similar to the containerd archives, and available from the code instead.

Or maybe not the go code, but to keep the nerdctl-full and the distro images in some kind of secondary config ?

---
"ubuntu:24.04":
  images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img"
    arch: "x86_64"
    digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img"
    arch: "aarch64"
    digest: "sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22"
  # Fallback to the latest release image.
  # Hint: run `limactl prune` to invalidate the cache
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
    arch: "aarch64"
---
"nerdctl-full:1.7.6":
  images:
  - location: "https://github.com/containerd/nerdctl/releases/download/1.7.6/nerdctl-full-1.7.6-linux-amd64.tar.gz"
    arch: "x86_64"
    digest: "sha256:2c841e097fcfb5a1760bd354b3778cb695b44cd01f9f271c17507dc4a0b25606"
  - location: "https://github.com/containerd/nerdctl/releases/download/1.7.6/nerdctl-full-1.7.6-linux-arm64.tar.gz"
    arch: "aarch64"
    digest: "sha256:77c747f09853ee3d229d77e8de0dd3c85622537d82be57433dc1fca4493bab95"
    # No arm-v7
    # No riscv64

And then expanded by lima.

EDIT: Added:

Now the only duplication is the default mounts... Hmm.

images:
- default
mounts:
- default

Provide standard values "default" and "none" for most options #2458

afbjorklund · 2024-09-01T10:11:43Z

The resulting JSON is supposed to be uploaded to some external URL under https://schemastore.org/

Perhaps hosted on the lima-vm.io site, or as a fallback loaded from the git repo directly (like cloud-config)

Then it can be used while editing the lima.yaml, even though it still needs to pass limactl validate.

But the hidden command and the make target is mostly for developers / while changing default.yaml

AkihiroSuda · 2024-10-09T05:37:38Z

Needs rebase

Makefile

AkihiroSuda

Thanks

AkihiroSuda · 2024-10-10T01:02:55Z

Sorry, needs rebase again

AkihiroSuda

Thanks

AkihiroSuda · 2024-10-15T06:31:14Z

Still needs another rebase

examples/default.yaml

pkg/limayaml/limayaml.go

AkihiroSuda · 2024-10-18T22:59:40Z

Needs rebase

Signed-off-by: Anders F Björklund <[email protected]>

Use empty strings for strings and document all the fields Allow null for pointers and maps with commented out keys Signed-off-by: Anders F Björklund <[email protected]>

The null is only used in the default.yaml as a placeholder When loaded into a string or array or map, it is "empty". Signed-off-by: Anders F Björklund <[email protected]>

Signed-off-by: Anders F Björklund <[email protected]>

afbjorklund · 2024-10-19T07:35:23Z

I left out the information about which arrays are null in the default template, I don't think anyone cares...

But left the "nullable" in, so that we don't get validation errors:

  examples/default.yaml::$.additionalDisks: None is not of type 'array'
  examples/default.yaml::$.mountTypesUnsupported: None is not of type 'array'
  examples/default.yaml::$.networks: None is not of type 'array'
  examples/default.yaml::$.caCerts.files: None is not of type 'array'
  examples/default.yaml::$.caCerts.certs: None is not of type 'array'

Otherwise they could commented out, and handled by the "omitempty"

"Codespell found one or more problems" ArchTypes ==> archetypes Signed-off-by: Anders F Björklund <[email protected]>

AkihiroSuda · 2024-10-19T15:50:54Z

cmd/limactl/genschema.go

+	}
+	_, err = fmt.Fprintln(cmd.OutOrStdout(), string(j))
+	return err
+}


Can we have a test? What happens when this is out of the sync with the YAML struct definition?

This comment was marked as outdated.

Sign in to view

afbjorklund force-pushed the schema branch from a9983ec to 372bc2a Compare May 1, 2024 06:57

afbjorklund force-pushed the schema branch from 372bc2a to a177cd8 Compare May 1, 2024 07:14

afbjorklund force-pushed the schema branch 3 times, most recently from 8522c7a to 7aa660e Compare May 5, 2024 10:47

afbjorklund mentioned this pull request May 10, 2024

Change cpuType default to null #2333

Merged

afbjorklund force-pushed the schema branch 2 times, most recently from 7a6dbd7 to 0a19046 Compare May 16, 2024 18:20

afbjorklund force-pushed the schema branch from 76351cc to 33221ec Compare June 16, 2024 11:14

afbjorklund force-pushed the schema branch from 33221ec to f8f49a6 Compare August 31, 2024 11:39

afbjorklund marked this pull request as ready for review September 1, 2024 10:04

AkihiroSuda added this to the v1.0 milestone Sep 2, 2024

AkihiroSuda requested a review from jandubois September 2, 2024 06:45

afbjorklund force-pushed the schema branch 4 times, most recently from 772fb9e to e0697d7 Compare September 28, 2024 12:00

afbjorklund force-pushed the schema branch from e0697d7 to 97a13bf Compare October 5, 2024 10:58

AkihiroSuda reviewed Oct 9, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

afbjorklund force-pushed the schema branch from 97a13bf to 7b0cf97 Compare October 9, 2024 07:04

This comment was marked as resolved.

Sign in to view

AkihiroSuda previously approved these changes Oct 9, 2024

View reviewed changes

afbjorklund dismissed AkihiroSuda’s stale review via 12c2b33 October 14, 2024 06:54

afbjorklund force-pushed the schema branch from 7b0cf97 to 12c2b33 Compare October 14, 2024 06:54

AkihiroSuda previously approved these changes Oct 15, 2024

View reviewed changes

jandubois reviewed Oct 15, 2024

View reviewed changes

examples/default.yaml Outdated Show resolved Hide resolved

pkg/limayaml/limayaml.go Outdated Show resolved Hide resolved

pkg/limayaml/limayaml.go Outdated Show resolved Hide resolved

afbjorklund dismissed AkihiroSuda’s stale review via a183c28 October 17, 2024 06:32

afbjorklund force-pushed the schema branch 2 times, most recently from a183c28 to 8f7d173 Compare October 17, 2024 06:38

afbjorklund added 6 commits October 19, 2024 09:14

Add command to generate jsonschema for limayaml

88c17d3

Signed-off-by: Anders F Björklund <[email protected]>

Improve the limayaml and default.yaml syntax

b89e5b6

Use empty strings for strings and document all the fields Allow null for pointers and maps with commented out keys Signed-off-by: Anders F Björklund <[email protected]>

Don't mention null as the builtin default

395fb58

The null is only used in the default.yaml as a placeholder When loaded into a string or array or map, it is "empty". Signed-off-by: Anders F Björklund <[email protected]>

Change mount mountPoint to string pointer

b204276

Signed-off-by: Anders F Björklund <[email protected]>

Allow Disk to use either name or struct

475f4c6

Signed-off-by: Anders F Björklund <[email protected]>

Add jsonschema enum for the property types

cfb63d8

Signed-off-by: Anders F Björklund <[email protected]>

afbjorklund force-pushed the schema branch from 8f7d173 to cfb63d8 Compare October 19, 2024 07:29

Allow using composed words, like Arch and Types

55224aa

"Codespell found one or more problems" ArchTypes ==> archetypes Signed-off-by: Anders F Björklund <[email protected]>

AkihiroSuda reviewed Oct 19, 2024

View reviewed changes

AkihiroSuda approved these changes Oct 19, 2024

View reviewed changes

AkihiroSuda merged commit bc7788b into lima-vm:master Oct 19, 2024
30 checks passed

AkihiroSuda mentioned this pull request Oct 19, 2024

jsonschema should have tests #2768

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add command to generate jsonschema for limayaml #2306

Add command to generate jsonschema for limayaml #2306

afbjorklund commented Apr 30, 2024 •

edited

Loading

This comment was marked as outdated.

mac2net commented Apr 30, 2024

afbjorklund commented Apr 30, 2024 •

edited

Loading

afbjorklund commented May 1, 2024 •

edited

Loading

afbjorklund commented May 1, 2024 •

edited

Loading

afbjorklund commented Sep 1, 2024

AkihiroSuda commented Oct 9, 2024

This comment was marked as resolved.

AkihiroSuda left a comment

AkihiroSuda commented Oct 10, 2024

AkihiroSuda left a comment

AkihiroSuda commented Oct 15, 2024

AkihiroSuda commented Oct 18, 2024

afbjorklund commented Oct 19, 2024

AkihiroSuda Oct 19, 2024

Add command to generate jsonschema for limayaml #2306

Add command to generate jsonschema for limayaml #2306

Conversation

afbjorklund commented Apr 30, 2024 • edited Loading

This comment was marked as outdated.

mac2net commented Apr 30, 2024

afbjorklund commented Apr 30, 2024 • edited Loading

afbjorklund commented May 1, 2024 • edited Loading

afbjorklund commented May 1, 2024 • edited Loading

afbjorklund commented Sep 1, 2024

AkihiroSuda commented Oct 9, 2024

This comment was marked as resolved.

AkihiroSuda left a comment

Choose a reason for hiding this comment

AkihiroSuda commented Oct 10, 2024

AkihiroSuda left a comment

Choose a reason for hiding this comment

AkihiroSuda commented Oct 15, 2024

AkihiroSuda commented Oct 18, 2024

afbjorklund commented Oct 19, 2024

AkihiroSuda Oct 19, 2024

Choose a reason for hiding this comment

afbjorklund commented Apr 30, 2024 •

edited

Loading

afbjorklund commented Apr 30, 2024 •

edited

Loading

afbjorklund commented May 1, 2024 •

edited

Loading

afbjorklund commented May 1, 2024 •

edited

Loading