Generalization for np.isnan

[AlexRodis]

Currently, if one tries to supply observations from a numpy.ndarray with both strings and floats to a pymc.Distribution errors are raised. It seems pymc internally relies on numpy.isnan to check for missing values, which will raise in the above example. If the intention is to check for nan only there is a slightly more general way of doing this:

import numpy as np
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
names = load_iris().target_names
y = y[:,None].astype(object)
for i in range(3):
   y[y==i]=names[i]
# Dataset of the form (X|y) with y replaced with strings
arr = np.concatenate([X, y], axis=1)
# np.isnan(arr) Will raise
# A more general way
# `np.nan != np.nan` nan isn't equal to itself
isnan = np.vectorize(lambda elem != elem)
isnan(arr)

[ricardoV94]

Why are you passing an array with strings to PyMC? Those won't be handled correctly regardless of the failure at the isnan check.

PyMC will ultimately convert the data to a PyMC TensorVariable which doesn't accept string dtypes.

[AlexRodis]

The use case I had in mind involves the BEST model and a project I'm working on. The idea is store a categorical variable (possibly) a string along with the actual data. The array gets re indexed so the strings dont get passed to pymc, they are merely a convenience used instead to know what to group by. The problem arises because even though only the float part of the array gets passed to the observed variable, because the dtype is still object when pymc calls np.isnan it raises

[AlexRodis]

No, because it doesn't mesh well with the over structure of the code base. Perhaps some more context would help. I encountered this problem as part of a project I'm working on a bayesian models library. Because I'll have to handle different rank tensors and to allow the user to use whichever tensor library they like, all data handling tasks are delegated elsewhere, prior to grouping (since grouping is model specific, whereas other preprocessing tasks are not). If np.isnan is really necessary I can get by, by forcing users to used pandas DataFrames, though it is a bit inconvenient

[ricardoV94]

I mean when you pass observed=x can't you write observed=x.astype(distribution.rv_type.dtype) or something like that? I assume most tensor libraries support astype method?

[AlexRodis]

Bit "hacky" but I suppose it would work. Out of curiosity, is there any specific reason pymc is committed to numpy.isnan ? The method seems pretty limited to me

[AlexRodis]

Also I don't think this will work with a tensorflow tensor object (although I'm not expert in tf)

[ricardoV94]

I don't think there was a big principled reason, that was just what worked out of the box with the input types that we officially support.