How to specify that InvalidResultException is expected result

[FrieseWoudloper]

I added model inputs and expected results to the Model Verification part of my PMML file. This works fine.

However, I would also like to test the model response for invalid model inputs. How do I do that?

I tried leaving the outputfield elements from the model verification row empty. I also tried removing them.

When I evaluate the model I keep getting an InvalidResultException and then the evaluation of the model halts.

What I would like to achieve, is that the invalid result is the expected result and that the evaluation of the model continues.
Is this possible?

[vruusmann]

I would also like to test the model response for invalid model inputs. How do I do that?

Your question is raised in relation to the o.j.e.Evaluator#verify() method (evaluates the content of the embedded ModelVerification element), right?

Today, the answer is that it's not supported, because the designers of the ModelVerification element did not foresee the need to assert that some verification data record is non-evaluatable (ie. raises an error). In fact, the whole concept of evaluation errors is under-specified.

I tried leaving the outputfield elements from the model verification row empty. I also tried removing them.

We can design and implement something ourselves, and if it proves to be successful, submit it to DMG.org's consideration.

The simplest design would be "If the evaluation of a data record is expected to fail with an error, then the data record must not define any expected target or output values".

Successful data record (inputs plus targets and outputs):

<row>
	<petal_x0020_length>1.4</petal_x0020_length>
	<petal_x0020_width>0.2</petal_x0020_width>
	<sepal_x0020_length>1.4</sepal_x0020_length>
	<sepal_x0020_width>0.2</sepal_x0020_width>
	<PredictClass>Iris-setosa</PredictClass>
	<Iris-setosa_x0020_Prob>0.62</Iris-setosa_x0020_Prob>
	<Iris-versicolor_x0020_Prob>0.30</Iris-versicolor_x0020_Prob>
	<Iris-virginica_x0020_Prob>0.08</Iris-virginica_x0020_Prob>
</row>

Failing data record (only inputs, no targets or outputs):

<row>
	<petal_x0020_length>1.4</petal_x0020_length>
	<petal_x0020_width>0.2</petal_x0020_width>
	<sepal_x0020_length>1.4</sepal_x0020_length>
	<sepal_x0020_width>0.2</sepal_x0020_width>
</row>

A more complicated design would involve extending the VerificationField element (http://dmg.org/pmml/v4-3/ModelVerification.html#xsdElement_VerificationFields) with boolean attribute that signals the "error" state.

[vruusmann]

The indication of a failing test should be rather generic (eg. a boolean flag), it must not link to a particular error class (eg. o.j.e.InvalidResultException) or error message text, because the latter vary between PMML engines, and are not stable in time even within a single PMML engine.

For example, the o.j.e.InvalidResultException has a confusing name, and will be renamed to something more appropriate. See #145

[vruusmann]

What I would like to achieve, is that the invalid result is the expected result and that the evaluation of the model continues.
Is this possible?

It's not possible using o.j.e.Evaluator#verify(), but it is somwhat possible if you work directly with the low-level batch integration testing API.

Workflow:

1. Create a org.jpmml.evaluator.testing.Batch object.
2. Run this batch object using the org.jpmml.evaluator.testing.BatchUtil#evaluate(Batch) utility method. If the evaluation was successful, then it returns an empty list. If the evaluation failed, then it returns a list of org.jpmml.evaluator.testing.Conflict objects.
3. Inspect the state of conflict objects. If there was a mismatch between expected and actual fields values, then the Conflict#getDifference() method returns a non-empty map difference object. However, if the evaluation failed with an exception, then the map difference object is unset, and you can query the root cause using the Conflict#getException() method.
4. Test if the exception is an instance of o.j.e.InvalidResultException.

The conflict object can store at most one exception. So, if there are multiple invalid input field values, it will only "capture" the first invalid value, and ignore the rest.

Ideas for improving the batch testing framework:

1. Create specialized o.j.e.Conflict subclasses for the "mismatching results" vs "an error happened" cases.
2. The "an error happened" conflict class should be able to store any number of exceptions (similar to how markup inspectors can hold any number of exceptions, and will re-raise the first of them).
3. The BatchUtil#evaluate(Batch) utility method should attempt to prepare all input values (if some input value is invalid, then report it, and continue with the next).

[vruusmann]

Just opened an JPMML-Evaluator 1.6.X development branch. That's a good opportunity to work on the identified "structural problems".

[vruusmann]

I've requested DMG.org to clarify their position on this:
http://mantis.dmg.org/view.php?id=273

[vruusmann]

Some more thoughts about #220 (comment)

The o.j.e.BatchUtil#evaluate(Batch) util should return a list of "status objects", which represent both successful and failed evaluations. In other words, there should be a 1-to-1 positional mappings between inputs and results.

Right now the integration testing facility only reports failures. The conflict object maintains an integer row number as the Conflict#id property, but it's rather cumbersome to align and track lines "manually" in application code.

[FrieseWoudloper]

This really sounds great, and exactly what I need! However, I'm not a Java developer and cannot write Java code myself. Please let me know if there is any other way that I can be of assistance.

[vruusmann]

This really sounds great, and exactly what I need!

I'll probably extend the JPMML-Evaluator 1.5.X development branch a bit, and implement this "if the verification data record does not specify any target or output values, then assume that it must raise an error" there.

All the remaining batch integration testing functionality then goes into the 1.6.X development branch, because there will be breaking API changes.

I'm not a Java developer and cannot write Java code myself

The integration testing suite will also be re-modularized and re-factored to enable no-code/low-code setups.

For example, there will be easy runners for asserting that CSV input + Python Pickle file == Expected CSV results, CSV input + R RDS file == Expected CSV results, etc. Right now there's some custom Java code involved.

Please let me know if there is any other way that I can be of assistance.

Simply keep raising thought-provoking issues.

Anything about real-life data science workflows, where (J)PMML seems like a good technical platform (something to the tune of "almost works now, but could be improved in such and such ways").