Typing Generic mutability methods compatible with class inheritence

[charles-paperman]

Dear all,

I have some typing issues that I don't understand how to do. I write a toy example to explain its substance:

from typing import (
    Hashable,
    Callable,
    Generic,
    TypeVar,
    Self,
    Iterable,
    Optional
)

G = TypeVar("G", bound="Hashable")
K = TypeVar("K", bound="Hashable")

class TypedList(list[G]):
    def __init__(self, input_data : Iterable[G]):
        super().__init__(input_data)

    def map(self, fct: Callable[[G], K]) : # I want this to be Self[K] but it doesn't exists
        cls = type(self)
        return cls((fct(e) for e in self)) # This can't type correctly

class OrderedList(TypedList[G]):
    def __init__(self, input_data : Iterable[G]):
        input_data = sorted(input_data, key=hash)
        super().__init__(input_data)


O : OrderedList[int] = OrderedList((2, 3, 1, 0))
O.map(str) # This should return an OrderedList[str]

The PEP about Self mentioned that Self shoudln't be refered with generic argument for readability and suggest to use the following construct if I understand correctly:

class TypedList(list[G]):
    def __init__(self, input_data : Iterable[G]):
        super().__init__(input_data)

    def map(self, fct: Callable[[G], K])  -> TypedList[K]:
        return TypedList(fct(e) for e in self))

This work well and typed well but doesn't propagate to subclass OrderedList and

O.map(str) # This is now a TypedList and not and OrderedList

If I want the best of both world I need to overload map into the subclass which I would really not do as this is just for the typing.

What is the optimal solution here?

ps: I checked with mypy only, maybe another Static typer would not complained using Self in the first version.

[Daverball]

This sort of thing requires higher kinded types, which are quite challenging to implement, especially once you consider composability with other typing features. There has been some discussion about allowing Self to be higher kinded in a limited scope, but no concrete proposals have come out of it so far, since it requires careful consideration about what happens in subclasses that no longer have a type var.

For now you just can't do this, so you will have to pick your poison. I think returning TypedList[K] is the best option, it forces you to override in subclasses if you need the container type to match, but it's at least never a wrong type, it's just not as precise as it could be.

In your toy example you at least would have the option to change map to a classmethod which accepts any iterable as the first parameter and uses G as the return type parameter:

https://mypy-play.net/?mypy=latest&python=3.12&gist=0bf8f698d226e9e732d8bc020d3902a2

from typing import (
    Hashable,
    Callable,
    Generic,
    TypeVar,
    Self,
    Iterable,
    Optional
)

G = TypeVar("G", bound="Hashable")
K = TypeVar("K", bound="Hashable")

class TypedList(list[G]):
    def __init__(self, input_data : Iterable[G]):
        super().__init__(input_data)

    @classmethod
    def map(cls, data: Iterable[K], fct: Callable[[K], G]) -> Self:
        return cls((fct(e) for e in data))

class OrderedList(TypedList[G]):
    def __init__(self, input_data : Iterable[G]):
        input_data = sorted(input_data, key=hash)
        super().__init__(input_data)


O: OrderedList[int] = OrderedList((2, 3, 1, 0))
OrderedList.map(O, str)  # this works like you want it to

[charles-paperman]

Thank you.

Indeed, the classmethod solution is rather elegant. It is better than having to type subclasses.

I hope some kind of higher order will be integrated at some point, but I could imagine how hard it is to do that.