Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

np.recarray is not properly converted #553

Open
laggvar opened this issue Sep 10, 2024 · 4 comments
Open

np.recarray is not properly converted #553

laggvar opened this issue Sep 10, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@laggvar
Copy link

laggvar commented Sep 10, 2024

Affects: PythonCall

Describe the bug
It seems that pyconvert(Any, ::<np.recarray>) incorrectly assumes that a recarray can be wrapped as a PyArray.

julia> np = pyimport("numpy");

julia> arr = np.recarray((2,2), dtype = @py([("A", "O"), ("B", "O")]))
Python:
rec.array([[(None, None), (None, None)],
           [(None, None), (None, None)]],
          dtype=[('A', 'O'), ('B', 'O')])

julia> pyconvert(Any, arr)
2×2 PyArray{NamedTuple{(:A, :B), Tuple{PythonCall.Wrap.UnsafePyObject, PythonCall.Wrap.UnsafePyObject}}, 2}:
 (A = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830), B = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830))  (A = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830), B = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830))
 (A = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830), B = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830))  (A = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830), B = UnsafePyObject(Ptr{PyObject} @0x00007ffc75ba4830))

A field name can not be accessed after conversion.

julia> arr.A
Python:
array([[None, None],
       [None, None]], dtype=object)

julia> pyconvert(Any, arr).A
ERROR: type PyArray has no field A
Stacktrace:
 [1] getproperty(x::PyArray{NamedTuple{(:A, :B), Tuple{PythonCall.Wrap.UnsafePyObject, PythonCall.Wrap.UnsafePyObject}}, 2, true, false, NamedTuple{(:A, :B), Tuple{PythonCall.Wrap.UnsafePyObject, PythonCall.Wrap.UnsafePyObject}}}, f::Symbol)
   @ Base .\Base.jl:37
 [2] top-level scope
   @ REPL[11]:1

If indexing before field access, it works, but it does not return a usable wrapper.

julia> pyconvert(Any, arr)[1]
(A = PythonCall.Wrap.UnsafePyObject(Ptr{PythonCall.C.PyObject} @0x00007ffc75ba4830), B = PythonCall.Wrap.UnsafePyObject(Ptr{PythonCall.C.PyObject} @0x00007ffc75ba4830))

julia> pyconvert(Any, arr)[1].A
PythonCall.Wrap.UnsafePyObject(Ptr{PythonCall.C.PyObject} @0x00007ffc75ba4830)

I think the expected behavior should be pyconvert(Any, ::<np.recarray>) returning something equivalent to a StructArray, i.e. foo[1].A and foo.A[1] are equivalent.

Environment:
Julia v1.9.3
PythonCall v0.9.23

@laggvar laggvar added the bug Something isn't working label Sep 10, 2024
@laggvar
Copy link
Author

laggvar commented Sep 10, 2024

Part of the issue seems related to https://github.com/JuliaPy/PythonCall.jl/blob/main/src/Convert/pyconvert.jl#L222, where python objects following array interfaces get special treatment. In this case, however, the object has more structure than just array structure, which gets lost.

@cjdoris
Copy link
Collaborator

cjdoris commented Sep 10, 2024

The conversion to PyArray doesn't actually lose any structure - the numpy array really is essentially just an array of named tuples. The difference is that numpy gives you a way to access the subarray corresponding to a single component of these names tuples and PyArray doesn't. No reason we couldn't support a similar interface.

@cjdoris
Copy link
Collaborator

cjdoris commented Sep 10, 2024

A bigger issue is the presence of UnsafePyObject in the wrapped array - those ideally would be Py instead.

@laggvar
Copy link
Author

laggvar commented Sep 10, 2024

Yes, the maybe the structure itself is not lost, but it ambiguous in a conversion python -> julia -> python; since both recarrays and plain arrays of named tuples would become the same thing in the end. In my usecase at least, there is a need distinguish these two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants