Contract YAML updates

[tombaeyens]

There are 2 potential contract YAML syntax updates to consider:

1. group checks by type and force single checks for basic types like eg

checks:
  schema: skip
  freshness:
    column: created_at
  row_count:
    detect_anomaly: true

columns:
  - name: id
    checks:
      missing_values:
      invalid_values:
        valid_values: ['M']
      duplicate_values:
      sql_expressions:
        - ...
      sql_queries:
        - ...
      recon:
        other_dataset: CSTMRS

2. model the metadata separate from the checks. moving check configurations from the central metadata.
   Goal is to create a separation for the metadata that should go to data discovery from the unit testing responsibility.

The validity configurations in contracts now reside in the checks

dataset: dim_customer

columns:
- name: last_name
  data_type: character varying
  checks:
  - type: no_invalid_values
    valid_regex: '^(?:[A-Z])$'

There is the alternative to separate the meta data from the check configuration. For example:

dataset: dim_customer

columns:
- name: id
  data_type: character varying
  duplicates: not allowed
  missing: not allowed
  valid_regex: '^(?:[A-Z])$'
  checks:
  - type: no_missing_values
  - type: no_invalid_values

The above contains the metadata:

  duplicates: not allowed
  missing: not allowed
  valid_regex: '^(?:[A-Z])$'

And this metadata is in this alternative separated from the checks that configure Soda.

This way, metadata can be pushed to a catalog without it being tied to checks. So you can indicate that a column is unique without actually checking for it. Those are 2 separate concerns.

In similar style, we could also consider the schema check to be configured as a check. Eg

dataset: dim_customer

columns:
- name: id
  ...

checks:
- type: schema

But in the schema case, currently I think the default should be that the schema is checked. I think many users will forget to include the check and assume it's checked. And then later be surprised if the check is not performed.

3. Improve variable declarations for better tool support eg

checks_filter:
  variables:
    START_TS:
      required: true
      format: timestamp
    END_TS:
      required: true
      format: timestamp
  sql_expression:  ${START_TS} < ${TABLE_PREFIX}.'created_at' < ${END_TS}

[tools-soda]

CLOUD-8016