Does not work with future annotations

[gasparyanartur]

A resolution error is raised when you make a promise using annotations.
It works if you stop importing the annotations package and use quotation marks.

The following code breaks.
If you comment out the first line, the code works.

from __future__ import annotations
from plum import dispatch


class Displacement:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    @dispatch
    def __add__(self, other: "Displacement") -> "Displacement":
        return Displacement(self.x + other.x, self.y + other.y)

    @dispatch
    def __add__(self, other: int) -> "Displacement":
        return Displacement(self.x + other, self.y + other)


if __name__ == "__main__":
    d1 = Displacement(1, 2)
    d2 = Displacement(3, 4)
    print(d1 + d2)    

Traceback (most recent call last):
  File "C:\Users\gaspa\dev\python\py-chess\src\geometry_copy.py", line 22, in <module>
    print(d1 + d2)
  File "plum\function.py", line 585, in plum.function._BoundFunction.__call__
  File "plum\function.py", line 509, in plum.function.Function.__call__
  File "plum\function.py", line 373, in plum.function.Function._resolve_pending_registrations
  File "C:\Users\gaspa\.virtualenvs\py-chess-YZvcAOQM\lib\site-packages\plum\type.py", line 436, in is_object
    return t.get_types() == (object,)
  File "C:\Users\gaspa\.virtualenvs\py-chess-YZvcAOQM\lib\site-packages\plum\type.py", line 259, in get_types
    return ptype(self.resolve()).get_types()
  File "C:\Users\gaspa\.virtualenvs\py-chess-YZvcAOQM\lib\site-packages\plum\resolvable.py", line 41, in resolve
    raise ResolutionError(f"Promise `{self!r}` was not kept.")
plum.resolvable.ResolutionError: Promise `ForwardReferencedType(name="'Displacement'")` was not kept.

Process finished with exit code 1

[wesselb]

Thanks, @gasparyanartur. This is something that will have to be fixed. I imagine that it shouldn't be too complicated to get right.

[seeM]

Hi @wesselb 👋 I've been looking at replacing fastcore's (core library powering fastai) multiple dispatch implementation with plum in this PR. We're still in the early prototyping phase but it looks promising - it's been a mostly seamless process thanks to the great design of your library! 😄

Unfortunately this issue would be a blocker, but I'm happy to try implement a fix. If you could perhaps provide some pointers it would be super helpful and very much appreciated!

[wesselb]

Hey @gasparyanartur and @seeM,

Since there is increased interest in this issue, I've taken a stab at supporting from future import __annotations__. The particular bug @gasparyanartur ran into was easily fixed, but, unfortunately, I ran into a whole range of other issues. I've added experimental support for from future import __annotations__ on the latest commit on master. @gasparyanartur and @seeM, would you be able to see how that works for you?

Currently, there is just one gotcha. On Python 3.7 and 3.8 (not 3.9), if you're using from future import __annotations__, you should not be using strings for forward references. This is because typing.get_type_hints will, for those Python versions, actually not resolve strings to the right references, but wrap things in another ForwardReference type. The whole point, of course, of using from future import __annotations__ is that things don't have to be wrapped in strings anymore, so I don't think this should necessarily be an issue. This is what I mean:

Yes:

from plum import dispatch


class A:
    @dispatch
    def test(self) -> "A":
        return self

Also yes:

from __future__ import annotations

from plum import dispatch


class A:
    @dispatch
    def test(self) -> A:
        return self

But not this:

from __future__ import annotations

from plum import dispatch


class A:
    @dispatch
    def test(self) -> "A":
        return self

[wesselb]

Ok, actually, it's actually pretty easy to specifically deal with patterns of the form

from __future__ import annotations

from plum import dispatch


class A:
    @dispatch
    def test(self) -> "A":
        return self

I've commit a change to master which offers support for this. More complicated cases like

from __future__ import annotations

from typing import Union

from plum import dispatch


class A:
    @dispatch
    def test(self) -> Union[None, "A"]:
        return self

won't work. In that case, you really must not use quotes (on Python 3.7 and 3.8—things are fine on 3.9):

from __future__ import annotations

from typing import Union

from plum import dispatch


class A:
    @dispatch
    def test(self) -> Union[None, A]:
        return self

[gasparyanartur]

Awesome, I'll take a look at it a bit later

[gasparyanartur]

Hey @wesselb,

I tried out the lastest version and it works for the use-case I specified! However, it still breaks when you try dispatching with multiple classes. You mention something in the README about using plum.type.PromisedType instead, so I'm not sure if this is intended behavior or not.

The following code breaks:

class ExampleClassPure:
    @dispatch
    def example_function(self, x: ExampleClassPure):
        ...

    @dispatch
    def example_function(self, x: int):
        ...

    @dispatch
    def example_function(self: ExampleClassMixed):
        ...


class ExampleClassMixed:
    @dispatch
    def example_function(self, x: "ExampleClassMixed"):
        ...

    @dispatch
    def example_function(self, x: int):
        ...

def test_forward_reference_with_different_class():
    c1 = ExampleClassPure()
    c2 = ExampleClassMixed()
    c1.example_function(c2)

E           plum.function.NotFoundLookupError: For function "example_function" of tests.dispatcher.test_future_annotations.ExampleClassPure, signature Signature(tests.dispatcher.test_future_annotations.ExampleClassPure, tests.dispatcher.test_future_annotations.ExampleClassMixed) could not be resolved.

..\..\plum\function.py:463: NotFoundLookupError

I've added this test along some other in my local fork. If you want, I could push it and make a pull request.

[wesselb]

Hey @gasparyanartur! Thanks for checking this! Would you be able to post the full example here?

E.g., the following works for me locally:

from __future__ import annotations

from plum import dispatch


class A:
    @dispatch
    def do_a(self, a: A) -> A:
        return a

    @dispatch
    def do_b(self, a: B) -> B:
        return a


class B:
    @dispatch
    def do_a(self, a: A) -> A:
        return a

    @dispatch
    def do_b(self, a: B) -> B:
        return a


a = A()
b = B()
a.do_a(a)
a.do_b(b)
b.do_a(a)
b.do_b(b)

I think that should be roughly along the lines of your example, right?

[gasparyanartur]

Did you mean to write do_a instead of do_b? In that case it is the same as my example. Regardless, it turns out there was a bug in my test. This code does work on my machine as well.

However, it does not work in my project for some reason.
The strange thing is, I've tried my absolute hardest, but I cannot seem to replicate the bug

Here is the code gist:

	
from __future__ import annotations

from abc import ABC

from plum import dispatch
from dataclasses import dataclass


@dataclass(frozen=True, order=True, unsafe_hash=False, match_args=True, eq=True)
class Vector(ABC):
    x: int
    y: int

    @property
    def tuple(self):
        return self.x, self.y


class Position(Vector):
    @property
    def algebraic(self):
        return f"{chr(ord('a') + self.x)}{self.y + 1}"

    @property
    def i(self) -> int:
        return self.y * 9 + self.x

    @staticmethod
    def from_index(i: int) -> Position:
        y: int = i // 9
        x: int = i - y
        return Position(x, y)

    def in_board(self) -> bool:
        return (0 <= self.x < 9) and (0 <= self.y < 9)

    def __eq__(self, other: Vector) -> bool:
        if not isinstance(other, Vector):
            raise TypeError(f"Could not compare {type(self)} with {type(other)}")

        elif not isinstance(other, Position):
            return False

        return self.x == other.x and self.y == other.y

    def __add__(self, other: Displacement) -> Position:
        return Position(self.x + other.x, self.y + other.y)

    @dispatch
    def __sub__(self, other: Position) -> Displacement:
        return Displacement(self.x - other.x, self.y - other.y)

    @dispatch
    def __sub__(self, other: Displacement) -> Position:
        return Position(self.x - other.x, self.y - other.y)


class Displacement(Vector):
    def __eq__(self, other: Vector) -> bool:
        if not isinstance(other, Vector):
            raise TypeError(f"Could not compare {type(self)} with {type(other)}")

        elif not isinstance(other, Displacement):
            return False

        return self.x == other.x and self.y == other.y

    def __add__(self, other: Position | Displacement) -> Position | Displacement:
        if isinstance(other, Position):
            return Position(self.x + other.x, self.y + other.y)

        elif isinstance(other, Displacement):
            return Displacement(self.x + other.x, self.y + other.y)

        else:
            raise TypeError(f"Could not perform subtraction between classes {type(self)} and {type(other)}.")

    def __sub__(self, other: Displacement) -> Displacement:
        return Displacement(self.x - other.x, self.y - other.y)

    def __mul__(self, other: Displacement | int) -> Displacement:
        if isinstance(other, Displacement):
            return Displacement(self.x * other.x, self.y * other.y)

        elif isinstance(other, int):
            return Displacement(self.x * other, self.y * other)

I've included the failing test in the comments. I've added dispatching to the __sub__ function in Position, but it does not seem to work.

When I try replicating the error using a different class, it seems to work though:

from __future__ import annotations

from abc import ABC
from dataclasses import dataclass

from plum import dispatch


@dataclass(frozen=True, order=True, unsafe_hash=False, match_args=True, eq=True)
class X(ABC):
    x: int
    y: int
    ...


class A(X):
    @dispatch
    def __add__(self, a: A) -> B:
        return B(self.x - a.x, self.y - a.y)

    @dispatch
    def __add__(self, a: B) -> A:
        return A(self.x - a.x, self.y - a.y)


class B(X):
    @dispatch
    def __add__(self, a: A) -> A:
        return a


def test():
    a = A(1, 2)
    b = B(1, 2)

    x = a + b
    assert isinstance(x, A)

    y = a + a
    assert isinstance(y, B)

as seen in this gist.

However, given the fact that I cannot replicate the bug, the error is most likely in my code.

[wesselb]

@gasparyanartur I've left a comment on the Gist. (The Gist seems to run for me.)

[gasparyanartur]

You were right, it seemed that an old version of plum-dispatch was cached, which made that specific test crash. I tried re-cloning the repository and that solved the issue for me. The dispatching works flawlessly now. Thank you for the help!

[wesselb]

@gasparyanartur Absolutely no problem at all. :) I'm glad that things work for you now!

I'm just reopening this issue until @seeM has also confirmed that things work on their side.

[gasparyanartur]

It seems I was too hasty as well. Turns out I was running my old branch, and that's why it was working. Reinstalling everything from scratch, I still get the same error. I used Pipenv with the following file:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
multipledispatch = "*"
plum-dispatch = "1.5.10"
pytest = "*"

[dev-packages]

[requires]
python_version = "3.10"

[gasparyanartur]

Could it be that the latest official version is not the one that was pushed to master?

[wesselb]

Note that I've not yet pushed a new version with the latest commit on master. However, since things do seem to be working, I'll do that now. Could you try again on version v1.5.11, which should be available shortly?

[gasparyanartur]

Indeed, it looks like that was the error all along. Installing the latest v1.5.11 does fix the issue, and this time I double checked to make sure I was running the right branch 😅. Thank you for the quick response! I'll let you close the thread when you feel it has been resolved.

[wesselb]

Nice! :) I'm glad to hear that resolves the issue. Thanks for working with me to confirm that everything is running smoothly!

[seeM]

Thanks so much for looking into this @wesselb 😄!

Unfortunately our use-case is still hitting the error when using plum.Function.dispatch. Here is a simplified reproducible example. It works when the annotations import is commented, but fails with ResolutionError: Promise ForwardReferencedType(name='int') was not kept otherwise:

from plum import dispatch
# from __future__ import annotations  # uncommenting breaks this snippet

@dispatch
def f(x): return 'object'
def g(x: int): return 'int'
f.dispatch(g)
assert f('a') == 'object'
assert f(0) == 'int'

We're using this functionality to allow users of our Transform class to dispatch their own methods on top of a default method (in our case, the identity function).

Sticking to the Dispatcher.__call__ interface works with future annotations, and we could probably wrangle our case to use it, but plum.Function.dispatch does seem like a neater fit. This works (and an equivalent implementation in our Transform class):

from plum import dispatch
from __future__ import annotations

@dispatch
def f(x): return 'object'
@dispatch
def f(x: int): return 'int'
assert f('a') == 'object'
assert f(0) == 'int'

[wesselb]

Hey @seeM! No problem at all.

Unfortunately, adding support for from __future__ import annotations will have to happen on a feature-by-feature basis, but, if you wouldn't mind, I'm happy to keep resolving them until your use case is fully supported.

I've released a new version v1.5.12 which also adds future annotations support for Function.dispatch. Would you be able to try how that version now works for you?

[seeM]

That's understandable! :)

Hmm it still doesn't seem to be working:

from __future__ import annotations
from plum import dispatch, __version__

assert __version__ == '1.5.12'
@dispatch
def f(x): return 'object'
def g(x: int): return 'int'
f.dispatch(g)
assert f('a') == 'object'
assert f(0) == 'int'

ResolutionError: Promise `ForwardReferencedType(name='int')` was not kept.

[wesselb]

Hmm, this is very strange. The snippet runs for me locally. Let me look into it. Which Python version are you running?

[wesselb]

Apparently the compiled extension breaks the check for whether future annotations are used or not. That's odd. Looking into fixing this now.

[wesselb]

@seeM, alright, attempt two 😅: I've pushed a new version v1.5.13 which I believe fixes the compilation issue. Would you be able to see how that one works for you?

[seeM]

@wesselb it worked 😄 🎉 Again, thanks so much for the support!

I have one more issue 🙈 one of these commits have broken functions with no __annotations__, e.g the following works for 1.5.10 but not 1.5.13:

from __future__ import annotations
import math
from plum import dispatch
f = dispatch(math.sqrt)
f(3)

AttributeError: 'builtin_function_or_method' object has no attribute '__annotations__'

(BTW I'm not sure why some functions don't have __annotations__ - perhaps those are compiled?)

[wesselb]

@seeM Nice! :)

I have one more issue 🙈

We'll get there! I've pushed another release v1.5.14 which fixes this particular issue. Hopefully this one does the trick!

[seeM]

@wesselb That's fixed, hit another one :S Reproducible example:

from __future__ import annotations
from plum import dispatch
class A: pass
def f(x: A): pass
g = dispatch(f)
h = dispatch(f)
a = A()
g(a)  # h(a) also errors

File ~/.pyenv/versions/3.8.9/lib/python3.8/site-packages/plum/function.py:546, in plum.function.Function.__call__()

File ~/.pyenv/versions/3.8.9/lib/python3.8/site-packages/plum/function.py:402, in plum.function.Function._resolve_pending_registrations()

File ~/.pyenv/versions/3.8.9/lib/python3.8/site-packages/plum/function.py:60, in plum.function.extract_signature()

File ~/.pyenv/versions/3.8.9/lib/python3.8/site-packages/plum/util.py:145, in unquote(x)
    136 def unquote(x):
    137     """Remove quotes from a string at the outermost level.
    138
    139     Args:
   (...)
    143         str: `x` but without quotes.
    144     """
--> 145     while len(x) >= 2 and x[0] == x[-1] and x[0] in {'"', "'"}:
    146         x = x[1:-1]
    147     return x

TypeError: object of type 'type' has no len()

[wesselb]

Ah, that's a silly one on my part... Fixed in v1.5.15.

[seeM]

That's fixed but hit another. This one's very strange:

from plum.function import extract_signature
class A:
    def f(self): pass
extract_signature(A().f, True)

AttributeError                            Traceback (most recent call last)
Input In [14], in <cell line: 4>()
      2 class A:
      3     def f(self): pass
----> 4 extract_signature(A().f, True)

File ~/.pyenv/versions/3.8.9/lib/python3.8/site-packages/plum/function.py:60, in plum.function.extract_signature()

AttributeError: 'method' object has no attribute '__annotations__'

[seeM]

Hmm sorry, that might not be the best example for this issue... Trying to find a better one

[seeM]

Ignore my last few messages, this last issue seems to be on our side. I think this is all resolved now. Thanks so much! 😄

[wesselb]

Ah, are you sure there's nothing weird going on with extract_signature?

I'm glad that everything seems to be working on your side now, though! :) I think we can close the issue for now. If you come across any other weird behaviour or bugs, please reopen the issue and I'll be on it.

[PhilipVinc]

@wesselb i didn't check if you did that, but maybe it would be a good idea to add all those cases above to the test suite?

[wesselb]

@PhilipVinc Definitely! I indeed did that along the way of fixing all the bugs. The tests are here.

[PhilipVinc]

Great! Sorry for bothering you with that!

[wesselb]

Not a problem at all! It's good to double check these things.

[seeM]

@wesselb my bad, there is something going on with extract_signature. Here's an example closer to our use-case:

from __future__ import annotations
from plum import Function

class A:
    @classmethod
    def create(cls): pass

pf = Function(lambda x: x)
pf.dispatch(A.create)
pf()

AttributeError: 'method' object has no attribute '__annotations__'

[wesselb]

@seeM Thanks! Will be solved in the next release.

[seeM]

Thanks so much, this is working well too!

[bariod]

I might've come across another bug on dispatched staticmethods using py 3.11 (works with py 3.9):

from __future__ import annotations

from plum import dispatch


class Test:
    @staticmethod
    @dispatch
    def a() -> Test:
        return Test()

    @staticmethod
    @dispatch
    def a(b) -> Test:
        print(b)
        return Test()


x = Test()
x.a()
x.a(1)

results in:

Traceback (most recent call last):
  File "scratch_1.py", line 6, in <module>
    class Test:
  File "scratch_1.py", line 7, in Test
    @staticmethod
     ^^^^^^^^^^^^
  File "..\Python3111\Lib\site-packages\plum\function.py", line 120, in __doc__
    self._resolve_pending_registrations()
  File "..\Python3111\Lib\site-packages\plum\function.py", line 246, in _resolve_pending_registrations
    signature = extract_signature(f, precedence=precedence)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "..\Python3111\Lib\site-packages\plum\signature.py", line 187, in extract_signature
    for k, v in typing.get_type_hints(f).items():
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "..\Python3111\Lib\typing.py", line 2339, in get_type_hints
    hints[name] = _eval_type(value, globalns, localns)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "..\Python3111\Lib\typing.py", line 359, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "..\Python3111\Lib\typing.py", line 854, in _evaluate
    eval(self.__forward_code__, globalns, localns),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
NameError: name 'Test' is not defined

Process finished with exit code 1

using py 3.11.1 and plum-dispatch 2.1.0

I'm assuming 3.11 should also be fully supported..?

[wesselb]

@bariod, you're right! You caught a subtle bug. The bug has the do with the behaviour of @staticmethod, which is different between 3.9 and 3.11. The latest release v2.1.1 should fix the above issue.