cue vet and json.Validate behave differently

[cueckoo]

Originally opened by @dinhxuanvu in cuelang/cue#842

What version of CUE are you using (cue version)?

$ cue version
cue version 0.3.0-beta.5

Does this issue reproduce with the latest release?

Yes

What did you do?

I have a cue file with some definitions and a JSON file that I would like to run validation against the constraints that are specified in that cue file. I use both cue CLI and cue library to perform this task.

schema.cue

#test: {
  name: !=""
  image: !=""
  ...
}
#test

test.json

{
    "image": "something",
}

What did you expect to see?

I would expect both cue CLI and library would behave the same in term of validation.

What did you see instead?

When I run cue vet test.json schema.cue, it will complain name: incomplete value !="" as expected. However, when I use Validate(b []byte, v cue.Value) from encoding/json in go code, there is no error.

I wonder if this is expected behavior or a bug.

[cueckoo]

Original reply by @verdverm in cuelang/cue#842 (comment)

This does not match what I see. To get the error you describe, you have to supply the -d "#test" flag and value.

Are you doing the equivalent in the Go version?

[cueckoo]

Original reply by @dinhxuanvu in cuelang/cue#842 (comment)

This does not match what I see. To get the error you describe, you have to supply the -d "#test" flag and value.

Are you doing the equivalent in the Go version?

Hey @verdverm. Thank you for your response. You're correct. For the schema.cue that I use for CLI, it does have #test at the end. I forgot to include that in the report (corrected it now). For the schema.cue for the golang code, I do not have that. For golang code, I use something along this line:

import 	"cuelang.org/go/encoding/json"
instance := load.Instances([]string{"."}, &load.Config{Dir: /path/to/schema,})
is, err := c.runtime.Build(instance)
v := instance.LookupDef(key) // key == "test"
err = json.Validate(b, v) // b == []byte of test.json

There is no error reported. However, if the test.son looks like this:

{
    "name": 1111,
    "image": "something"
}

It will error out: name: conflicting values !=\"\" and 1111 (mismatched types string and int) as expected.
I'm a bit confused on why it will error out on the type mismatch correctly but not the missing field.
P/S: I use [email protected] for golang code.

[cueckoo]

Original reply by @verdverm in cuelang/cue#842 (comment)

You probably want to do something like:

topVal := inst.Value()
testVal := topVal.LookupPath(cue.ParsePath("#test"))
// check for errors / exists
// json.Validate

I suspect that you have an non-existent value after your lookup. You could also try making the key #test with the #, that may work. Either way, you should be checking the returned value for errors / existence.

[cueckoo]

Original reply by @dinhxuanvu in cuelang/cue#842 (comment)

I vendored v0.3.0-beta6 and I tested my code along with Tony's suggestion as well. The result is the same. No error on !="" check when the field is missing.

[cueckoo]

Original reply by @myitcv in cuelang/cue#842 (comment)

Please can you provide a complete example of what it is you are trying here, with expected vs actual output/behaviour? For some tips/tricks on creating good reproducers please see https://github.com/cuelang/cue/wiki/Creating-test-or-performance-reproducers

[cueckoo]

Original reply by @dinhxuanvu in cuelang/cue#842 (comment)

Hey Paul,

What I'm trying to do is validating a JSON against constraints that are specified in a definition in cuelang. As I mentioned on the on the description, I have a cue file schema.cue:

#test: {
  name: !=""
  image: !=""
  ...
}
#test

a JSON file test.json:

{
    "image": "something",
}

If you run cue vet test.json schema.cue, it will report an error about the name field is missing/non-empty. This is expected behavior because the schema for #test requires name field to be non-empty. However, if you use the Validate() function in encoding/json package to perform the same validation process, the test.json is considered valid against the schema and passes through the validation without any error. There is a discrepancy between the CLI validation and the json validation.
Plus, if you convert that test.json to YAML format test.yaml and use Validate() on encoding/yaml, then test.yaml will fail validation with the same error that cue vet will report. So you can see the same function that should perform the same way between JSON and YAML, somehow perform differently given the same input. As I mentioned in the PR, it seems to me the root cause is json.Validate() doesn't do concrete validation unlike yaml and cue vet.
I think the cue CLI part is clean to reproduce but perhaps the golang part requires some actual code. I can write some test code and put it a repo somewhere for you to see if you still feel this is not clear enough.

[cueckoo]

Original reply by @myitcv in cuelang/cue#842 (comment)

I can write some test code and put it a repo somewhere for you to see if you still feel this is not clear enough.

Yes please: it will help us all be clear we are reproducing the same issue. The advantage of such a repro is that it will, assuming we agree there is a bug, form the basis of a test case.

You don't necessarily need to push this code to a repo anywhere; the wiki I linked to shows how a standalone txtar repro can be created. But please don't feel you have to do things that way, it's just a suggestion.

[cueckoo]

Original reply by @dinhxuanvu in cuelang/cue#842 (comment)

I can write some test code and put it a repo somewhere for you to see if you still feel this is not clear enough.

Yes please: it will help us all be clear we are reproducing the same issue. The advantage of such a repro is that it will, assuming we agree there is a bug, form the basis of a test case.

You don't necessarily need to push this code to a repo anywhere; the wiki I linked to shows how a standalone txtar repro can be created. But please don't feel you have to do things that way, it's just a suggestion.

I think writing some code in a repo is probably easier for me. I'll try to get something up soon and will tag you here when it is ready.

[cueckoo]

Original reply by @myitcv in cuelang/cue#842 (comment)

@dinhxuanvu I've migrated this to be a discussion because I'm not clear there's an issue/bug right now, given what's described above. We can of course create issues for bugs/other that come from this discussion.

[cueckoo]

Original reply by @mpvl in cuelang/cue#842 (comment)

See also the #822. In general, people seem to want the behavior of ignoring inconcrete values when exporting, and instead be explicit if a value should be required. This is because in large APIs most fields tend to be optional.