Type mismatch for variadic generics variable in class __init__ VS method contexts

[toinbis]

(Cross-posting from https://stackoverflow.com/questions/76943605/type-mismatch-for-variadic-generics-variable-in-class-init-vs-method-context)

I have a type mismatch between the same definition of variadic generic Union[*HandlerTypes] used in __init__ and in pet_handler method.

I want strongly typed decorators - I achieve this by using TypeVarTuple variadic generics. All seems to work. But there is one last problem - the type mismatch between:

a) self.handlers: list[Union[*HandlerTypes]] = []
b) self.handlers.append(handler_func)

Though technically the type is the same Union[*HandlerTypes] appearantly the difference of the usage context (class init vs method) triggers the pylance error for self.handlers.append(handler_func):

Argument of type "Union[*HandlerTypes@AppModule]" cannot be assigned to parameter "__object" of type "*HandlerTypes@AppModule" in function "append"
  Type "Union[*HandlerTypes@AppModule]" cannot be assigned to type "*HandlerTypes@AppModule"PylancereportGeneralTypeIssues

Any ideas how I could work this out? Thanks a lot in advance - hours count spent on this is closely getting to 100 :)

Full code below:

class Dog:
    pass

class Cat:
    pass

class MyFirstDog(Dog):
    pass

class MyFirstCat(Cat):
    pass

class MySecondDog(Dog):
    pass

class MySecondCat(Cat):
    pass

HandlerType1 = Callable[[MyFirstDog, MyFirstCat], Tuple[MyFirstDog, MyFirstCat]]
HandlerType2 = Callable[[MySecondDog, MySecondCat], Tuple[MySecondDog, MySecondCat]]

HandlerTypes = TypeVarTuple('HandlerTypes')

class AppModule(Generic[*HandlerTypes]):
    def __init__(self) -> None:
        self.handlers: list[Union[*HandlerTypes]] = []

    def pet_handler(self, handler_func: Union[*HandlerTypes]) -> Union[*HandlerTypes]:
        self.handlers.append(handler_func)
        return handler_func

appmodule = AppModule[HandlerType1, HandlerType2]()

@appmodule.pet_handler
def pet_handler1(a: MyFirstDog, b: MyFirstCat) -> Tuple[MyFirstDog, MyFirstCat]:
    return (a, b)

@appmodule.pet_handler
def pet_handler2(a: MySecondDog, b: MySecondCat) -> Tuple[MySecondDog, MySecondCat]:
    return (a, b)

[Daverball]

This just looks like a bug in pylance to me, considering it thinks the first argument to append should be *HandlerTypes rather than Union[*HandlerTypes]. So I'd suggest opening a bug report for this.

[Daverball]

But I'm also honestly not sure if this is even supposed to be legal. I don't think Union[*Ts] was part of PEP646.

[Daverball]

Also I don't think you need variadic generics for this, why don't you just make a regular Generic with a single TypeVar and bind that to Union[HandlerType1, HandlerType2] instead? Do you ever unpack the TypeVarTuple into anything other than a Union?

[toinbis]

Thanks a lot for the comment @Daverball !

As per your recommendation, I managed to get a result without using variadic generics:

from typing import Tuple, Generic, Callable, Union, TypeVar

class Dog:
    pass

class Cat:
    pass

class MyFirstDog(Dog):
    pass

class MyFirstCat(Cat):
    pass

class MySecondDog(Dog):
    pass

class MySecondCat(Cat):
    pass

HandlerType1 = Callable[[MyFirstDog, MyFirstCat], Tuple[MyFirstDog, MyFirstCat]]
HandlerType2 = Callable[[MySecondDog, MySecondCat], Tuple[MySecondDog, MySecondCat]]
QueryHandlerList = Union[HandlerType1, HandlerType2]

TQueryHandler = TypeVar('TQueryHandler', bound=QueryHandlerList)

class AppModule(Generic[TQueryHandler]):
    def __init__(self) -> None:
        self.handlers: list[TQueryHandler] = []

    def pet_handler(self, handler_func: TQueryHandler) -> TQueryHandler:
        self.handlers.append(handler_func)
        return handler_func

appmodule = AppModule[QueryHandlerList]()

@appmodule.pet_handler
def pet_handler1(a: MyFirstDog, b: MyFirstCat) -> Tuple[MyFirstDog, MyFirstCat]:
    return (a, b)

@appmodule.pet_handler
def pet_handler2(a: MySecondDog, b: MySecondCat) -> Tuple[MySecondDog, MySecondCat]:
    return (a, b)

Thanks a ton!

As for the legality of Union[*Ts] - I find type unions may not appear within the Tuple in https://peps.python.org/pep-0646/, which probably means what you are suggesting that my construct was illegal in the first place.

I guess it's safe to close this discussion. Thanks again @Daverball for saving me! :)

[erictraut]

@toinbis, your initial code sample in this thread should type check without errors. This is a bug in pyright (the type checker upon which pylance is built). Union[*Ts] is supported in PEP 646, although it was not easy to implement. It requires special-casing (at least in the pyright implementation), and I'm not surprised that I missed one of the special cases. I'd appreciate it if you'd file a bug in the pyright issue tracker. I should be able to get a fix in place quickly.

[toinbis]

@erictraut - thanks a lot for implementing the feature in the first place (and building pyright - it's great)! And also for thanks this comment which clarified things for me. I have filed in the bug report - microsoft/pyright#5775, am glad to clarify things there if need. Thanks again!

[toinbis]

It turns out i've celebrated too early :/

What I missed that for simplicity reasons I've put the example in a single file. Yet in reality I want to achieve something like:

application.py:

from typing import Tuple, Callable, Union
from .myframework import AppModule, Dog, Cat


class MyFirstDog(Dog):
    pass

class MyFirstCat(Cat):
    pass

class MySecondDog(Dog):
    pass

class MySecondCat(Cat):
    pass

HandlerType1 = Callable[[MyFirstDog, MyFirstCat], Tuple[MyFirstDog, MyFirstCat]]
HandlerType2 = Callable[[MySecondDog, MySecondCat], Tuple[MySecondDog, MySecondCat]]
QueryHandlerList = Union[HandlerType1, HandlerType2]


appmodule = AppModule[QueryHandlerList]()

@appmodule.pet_handler
def pet_handler1(a: MyFirstDog, b: MyFirstCat) -> Tuple[MyFirstDog, MyFirstCat]:
    return (a, b)

@appmodule.pet_handler
def pet_handler2(a: MySecondDog, b: MySecondCat) -> Tuple[MySecondDog, MySecondCat]:
    return (a, b)

framework.py:

from typing import Generic, TypeVarTuple, Union

class Dog:
    pass

class Cat:
    pass


HandlerTypes = TypeVarTuple('HandlerTypes')


class AppModule(Generic[*HandlerTypes]):
    def __init__(self) -> None:
        self.handlers: list[Union[*HandlerTypes]] = []

    def pet_handler(self, handler_func: Union[*HandlerTypes]) -> Union[*HandlerTypes]:
        self.handlers.append(handler_func) # type: ignore
        return handler_func

As the name implies - framework.py is something I am writing in advance without knowing what will be passedd to AppModule during initialisation. Therefore I believe variadic generics do fit here at the end of the day.

The code as it is (with self.handlers.append(handler_func) # type: ignore) works without pyright errors.
As I have spend that much time to get to the solution, I'd appreciate if we'd keep this discussion a live for a bit more just to clarify everything till the end. Namely:

a) Is Union[*HandlerTypes] really an invalid construct? I lean to believe it is, but seeing pylance not complain about it makes me wonder.
b) Isn't it so that the Pyright/pylance should complain about this construct - should I be opening a ticket in any case: pylance is happy with usage in once context, shows error in another.
c) Maybe there still is a way to make this work without type: ignore? Am sure there is similar magic to #1347 (comment) out there and I'm sure I wont discover it myself.
d) And if the only way to solve this is 'accept and move one' - where would you cut corners, leave as it is with type:ignore or maybe go with any other shorcut?

Thanks a lot. @Daverball - maybe you'll be up to reading my update. Thanks again!

I've found three related tickets:
#1442 - [BUG] Union should expand TypeVarTuple
#1347 - Type-hint instances of types passed as TypeVarTuple
#1301 (comment) - Callable with variadic fixed arguments that doesn't mean *args

[Daverball]

Here's a little mypy-play to show that this works: https://mypy-play.net/?mypy=latest&python=3.11&gist=83149770b1f46b30f63a7c661604eb67

[toinbis]

@Daverball - thanks a log again!

a) https://mypy-play.net/?mypy=latest&python=3.11 - this seems to be a generic link to mypy-play, not a link to a specific example which you have crafted (for which I'm very grateful)!

b) Your example is clear, expressive and works out of the box! Most likely i'll end up using it! Time to learn Protocol :)

c) Yet I was up for a mission to find a way for it to work without Any :) (Promise, last time I jump into such rabbit hole!). Which brings me to attempt to type def __call__(self, __a: Any, __b: Any) -> tuple[Any, Any]: ... more strictly:

# myframework.py

from typing import Generic, TypeVar, Protocol, Tuple

# Base classes
class Dog:
    pass

class Cat:
    pass

# Define a generic protocol for the handler functions
DogType = TypeVar('DogType', bound=Dog)
CatType = TypeVar('CatType', bound=Cat)

class Handler(Protocol[DogType, CatType]):
    def __call__(self, __a: DogType, __b: CatType) -> Tuple[DogType, CatType]: ...

HandlerType = TypeVar('HandlerType', bound=Handler[Dog, Cat])

class AppModule(Generic[HandlerType]):
    def __init__(self) -> None:
        self.handlers: list[HandlerType] = []

    def pet_handler(self, handler_func: HandlerType) -> HandlerType:
        self.handlers.append(handler_func)
        return handler_func

and

# application.py (nothing changed since last example)

from typing import Tuple, Callable, Union
from myframework import AppModule, Dog, Cat

# Subclasses
class MyFirstDog(Dog):
    pass

class MyFirstCat(Cat):
    pass

class MySecondDog(Dog):
    pass

class MySecondCat(Cat):
    pass

# Handlers types
HandlerType1 = Callable[[MyFirstDog, MyFirstCat], Tuple[MyFirstDog, MyFirstCat]]
HandlerType2 = Callable[[MySecondDog, MySecondCat], Tuple[MySecondDog, MySecondCat]]
QueryHandlerList = Union[HandlerType1, HandlerType2]

appmodule = AppModule[QueryHandlerList]()

@appmodule.pet_handler
def pet_handler1(a: MyFirstDog, b: MyFirstCat) -> Tuple[MyFirstDog, MyFirstCat]:
    return (a, b)

@appmodule.pet_handler
def pet_handler2(a: MySecondDog, b: MySecondCat) -> Tuple[MySecondDog, MySecondCat]:
    return (a, b)

Now the only error I get is for appmodule = AppModule[QueryHandlerList]() and reads as Type "QueryHandlerList" cannot be assigned to type variable "HandlerType@AppModule" Type "QueryHandlerList" is incompatible with bound type "Handler[Dog, Cat]" for type variable "HandlerType@AppModule" Type "QueryHandlerList" cannot be assigned to type "Handler[Dog, Cat]" Type "HandlerType1" cannot be assigned to type "(__a: Dog, __b: Cat, /) -> Tuple[Dog, Cat]" Parameter 1: type "Dog" cannot be assigned to type "MyFirstDog" "Dog" is incompatible with "MyFirstDog" Parameter 2: type "Cat" cannot be assigned to type "MyFirstCat" "Cat" is incompatible with "MyFirstCat"Pylance[reportGeneralTypeIssues](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportGeneralTypeIssues)

I understand it's a standard typing situation - I am trying to assign subtype variable to parent type (MyFirstDog vs Dog).

d) Which brings me to (really hope that last) two questions:

1. Would using Covariant TypeVars help me to get where I want witout Any and without erros? (not that I necessarily intend to go down that path). Any other means to achieve same? Are there other ways to say handler "you will get subtype of Dog"?
2. Maybe there is some rule of thumb I'm braking, i.e. "if you end up in a situation where you are trying to match subtype and type - don't bother, use Any and move in".

Thanks a lot for any thoughts!!

[Daverball]

I fixed the mypy-play link.

Variance gets a bit complicated once you're using callback protocols, the issue is that since you're using DogType and CatType as input arguments as well as return values they can neither be covariant, nor contravariant, they have to be invariant. So Any and an unbound TypeVar are the only way to get around this (so the type solver can pick a more specific type signature), but since you are using Handler as an upper bound for HandlerType you can't use an unbound TypeVar either, so Any is the only real option you have, unless you make AppModule generic in DogType and CatType and pass that forward to a generic protocol which is variant in DogType and CatType as well, but I'm not sure how well that would interact with type unions.

[Daverball]

So in short HandlerType = TypeVar('HandlerType', bound=Handler[Any, Any]) should work and will convey intent better about what a Handler is but otherwise will not be that much more type safe, than if you had specified Any for all the arguments and return values.

[toinbis]

@Daverball - wow, what great comments! Thank you - it is much appreciatated! All the geeky karma point's I've just earned for discovering and reporting pyright bug (with variadic generics typing) - I'm forwarding them to you! It is just what I needed - birds-eye view on the situation in my rabbit hole :) Super insightful.

Will be trying to finalise this part of my application in the comming days. Am pretty sure i'll end up using HandlerType = TypeVar('HandlerType', bound=Handler[Any, Any]), but for the sake of interest will also try using variadic generics. One obvious minus of variadic generics is that I want to use separate decorators for registering Queries and and Commands, and Application class can only accept a single variadic generic.

Will write an update here when I finish and will then close the discussion. Thanks a bunch again for all knowledge shared!

[toinbis]

Implemented using

class Handler(Protocol):
    def __call__(self, __a: Any, __b: Any) -> tuple[Any, Any]: ...

Works neat. Closing the topic, thanks @Daverball again!!