Implement DataValidator.apply()

[danielhuppmann]

This is the implementation for the apply method of the DataValidator.

The current implementation writes each validation-item to the log separately, together with the failing data rows.

2024-08-19 10:10:33 ERROR    Failed data validation (file validate_data/validate_data_fails.yaml):
  Criteria: variable: ['Primary Energy'], upper_bound: 5.0
         model scenario region        variable   unit  year  value
    0  model_a   scen_a  World  Primary Energy  EJ/yr  2010    6.0
    1  model_a   scen_b  World  Primary Energy  EJ/yr  2010    7.0

  Criteria: variable: ['Primary Energy|Coal'], lower_bound: 2.0
         model scenario region             variable   unit  year  value
    0  model_a   scen_a  World  Primary Energy|Coal  EJ/yr  2005    0.5

  Criteria: variable: ['Primary Energy'], year: [2005], upper_bound: 1.9, lower_bound: 1.1
         model scenario region        variable   unit  year  value
    0  model_a   scen_a  World  Primary Energy  EJ/yr  2005    1.0
    1  model_a   scen_b  World  Primary Energy  EJ/yr  2005    2.0

I initially had all failing validations (per yaml file) concatenated to one DataFrame, but I then thought that it would not be helpful if users can't see which criteria they specifically do not pass.

If you agree with this approach in principle, I can make the error-message a bit nicer (indentation, more concise criteria representation).

[phackstock]

Looks good to me, some smaller comments in line and one file to delete (I think).
After that, good to merge.

[phackstock]

You could turn this into a single list comprehension and use the walrus operator:

if error_list := [
                "  Criteria: "
                + ", ".join([f"{key}: {value}" for key, value in item.criteria.items()])
                + "\n"
                + textwrap.indent(str(df.validate(**item.criteria)), prefix="    ")
                + "\n"
                for item in self.criteria_items
                if df.validate(**item.criteria) is not None
            ]:
                logger.error(
                    "Failed data validation (file %s):\n%s",
                    get_relative_path(self.file),
                    "\n".join(error_list),
                )

not sure if that's more readable though.
Feel free to keep whatever is most readable to you.

[danielhuppmann]

This seems less readable than the current implementation, so suggest to get it unless we make a utility-function that moves the string-concat somewhere else - maybe together with a refactor of the RequiredDataValidator?