pyconvert_add_rule() ignored if cache has been populated

[hhaensel]

Currently, if I define a new rule from the beginning, everything works fine:

using PythonCall
using DataFrames

pdf = pytable(DataFrame(:a => 1:2, :b => 2:3))

PythonCall.pyconvert_rule_pandasdataframe(::Type{DataFrame}, x::Py) = DataFrame(PyTable(x))
PythonCall.pyconvert_add_rule("pandas.core.frame:DataFrame", DataFrame, PythonCall.pyconvert_rule_pandasdataframe, PythonCall.PYCONVERT_PRIORITY_ARRAY)

pyconvert(Any, pdf)

results in

2×2 DataFrame       
 Row │ a      b     
     │ Int64  Int64 
─────┼──────────────
   1 │     1      2
   2 │     2      3

However, if I do conversion once before the rule is defined, the conversion rules are cached and no conversion is performed. So far that's expected. But even if I delete the rule cache, the new rule is not applied.

using PythonCall
using DataFrames

pdf = pytable(DataFrame(:a => 1:2, :b => 2:3))

pyconvert(Any, pdf)

PythonCall.pyconvert_rule_pandasdataframe(::Type{DataFrame}, x::Py) = DataFrame(PyTable(x))
PythonCall.pyconvert_add_rule("pandas.core.frame:DataFrame", DataFrame, PythonCall.pyconvert_rule_pandasdataframe, PythonCall.PYCONVERT_PRIORITY_ARRAY)

empty!(PythonCall.PYCONVERT_RULES_CACHE)

pyconvert(Any, pdf)

results in

2×2 PyPandasDataFrame
   a  b
0  1  2
1  2  3

The reason is that somehow @generated pyconvert_rules_cache does not recalculate the value after the cache has been emptied.

julia> PythonCall.pyconvert_rules_cache(Any)
Dict{Ptr{PythonCall.C.PyObject}, Vector{Function}} with 1 entry:
  Ptr{PyObject} @0x00000219a6cf69e0 => [#54, #54, #54, #54, #54, #54, #54, #54, #54, #54, #54]

If I omit the @generated in front of pyconvert_rules_cache, the new rules are considered.

If this is by design in order to gain performance, it's perhaps worth mentioning in the docs of pyconvert_add_rule(). Otherwise removing @generated in convert.jl could be a solution.

[hhaensel]

Another posibility would be offering a resetting function

function pyconvert_reset_cache!()
    empty!(PYCONVERT_RULES_CACHE)
    Core.eval(@__MODULE__, quote
        @generated pyconvert_rules_cache(::Type{T}) where {T} = get!(Dict{C.PyPtr, Vector{Function}}, PYCONVERT_RULES_CACHE, T)
    end)
end

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[cjdoris]

Hi, I'll take a look. AFAIR calling pyconvert_add_rule should reset caches already, but maybe one of the later-stage caches is not being cleared.

[hhaensel]

Hi, just read that you are looking into this.
Meanwhile I have defined two functions for myself:

function pyconvert_reset_cache!()
    empty!(PythonCall.PYCONVERT_RULES_CACHE)
    Core.eval(PythonCall, quote
        @generated pyconvert_rules_cache(::Type{T}) where {T} = get!(Dict{C.PyPtr, Vector{Function}}, PYCONVERT_RULES_CACHE, T)
    end)
end

function pyconvert_reset!()
    empty!(PythonCall.PYCONVERT_RULES)
    empty!(PythonCall.PYCONVERT_EXTRATYPES)
    pyconvert_reset_cache!()
    PythonCall.init_pyconvert()
end