WIP: Add support for NaN and mixed types in infer_labels()

[hagenw]

Closes #12

This adds support for having NaN in truth or prediction and for having mixed types like ['a', 0] in truth and prediction.

[codecov]

Codecov Report

Merging #13 (513244d) into master (279806b) will not change coverage.
The diff coverage is 100.0%.

Impacted Files	Coverage Δ
audmetric/core/utils.py	100.0% <100.0%> (ø)

[frankenjoe]

I see the use-case for supporting NaN. But does it make sense to have mixed types? Isn't that rather a sign for mis-use? E.g. the user passes strings for truth but ids for prediction. I wonder if it makes more sense to raise an error in that case.

[frankenjoe]

Btw: how do we treat NaN in our metrics? E.g. how do we account for those labels in a confusion matrix?

[hagenw]

Btw: how do we treat NaN in our metrics? E.g. how do we account for those labels in a confusion matrix?

We have not really thought much about it. For some it just works:

>>> audmetric.unweighted_average_recall(['a', 'b'], ['a', 'a'])
0.5
>>> audmetric.unweighted_average_recall(['a', 'b'], ['a', np.NaN])
0.5

Others did strange stuff in master:

>>> audmetric.precision_per_class([0, 0, 2, 1], [0, 1, np.NaN, np.NaN])
{0: 1.0, 1: 0.0, 2: 0.0, nan: 0.0}

Now it would look like this

>>> audmetric.precision_per_class([0, 0, 2, 1], [0, 1, np.NaN, np.NaN])
{0: 1.0, 1: 0.0, 2: 0.0}

[hagenw]

I see the use-case for supporting NaN. But does it make sense to have mixed types? Isn't that rather a sign for mis-use? E.g. the user passes strings for truth but ids for prediction. I wonder if it makes more sense to raise an error in that case.

I'm not so sure about this. If you have audformat with its fixed types in mind, maybe. But I would find it valid if for example the labels are IDs collected by users, and some decide to enter an integer and others to enter a string.

[frankenjoe]

I'm not so sure about this. If you have audformat with its fixed types in mind, maybe. But I would find it valid if for example the labels are IDs collected by users, and some decide to enter an integer and others to enter a string.

But how do you map between strings and integers? Or do you assume they represent different classes? Why would a user do that? I still see the risk that integer and strings are accidentally mixed and the user will not notice it if allow that.

[frankenjoe]

We have not really thought much about it. For some it just works:

I guess we either have to come up with a proper solution or otherwise I think it would be safer to raise an error if we encounter NaN.

[hagenw]

OK, I created #14 and set this pull request to WIP.