How to annotate generic class with ParamSpec argument in older versions

[bryanforbes]

Let's say I have something along these lines (this is completely contrived) and I'm using Python 3.9:

from collections.abc import Callable
from typing import Generic, TypeVar
from typing_extensions import ParamSpec

P = ParamSpec('P')


class Invoker(Generic[P]):
    __slots__ = ('callback',)

    def __init__(self, callback: Callable[P, bool], /) -> None:
        self.callback: Callable[P, bool] = callback

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> bool:
        return self.callback(*args, **kwargs)


def compare_ints(a: int, b: int) -> bool:
    return a == b


def compare_floats(a: float, b: float) -> bool:
    return a == b


ints_invoker = Invoker(compare_ints)  # Invoker[(a: int, b: int)]
floats_invoker = Invoker(compare_floats)  # Invoker[(a: float, b: float)]


class IntsInvoker(Invoker[[int, int]]):
    ...


ints_invoker = IntsInvoker(compare_ints)
floats_invoker = IntsInvoker(compare_floats)  # error in type checker, if it is supported

According to PEP-612, the above should be valid. However, if I run this using Python 3.9 or lower, it will crash with:

TypeError: Parameters to generic types must be types. Got [<class 'int'>, <class 'int'>].

I should also be able to define IntsInvoker in this way:

class IntsInvoker(Invoker[int, int]):
    ...

However that also crashes using Python 3.9 or lower:

TypeError: Too many parameters for <class '__main__.Invoker'>; actual 2, expected 1

What is the proper way to define IntsInvoker so it will work in Python 3.9 and 3.10?

[srittau]

This line:

class IntsInvoker(Invoker[[int, int]]):

must be spelled like this:

class IntsInvoker(Invoker[tuple[int, int]]):

[int, int] is not a valid type. Invoker[int, int] is also incorrect, since Invoker has only type argument, but you provide two.

[bryanforbes]

According to PEP-612 the following is allowed:

P = ParamSpec('P')

class Z(Generic[P]):
  f: Callable[P, int]

def f(x: Z[[int, str, bool]]) -> str: ...   # Accepted
def f(x: Z[int, str, bool]) -> str: ...     # Equivalent

In Python 3.10, I can do something similar:

from collections.abc import Callable
from typing import Generic, ParamSpec, get_type_hints

P = ParamSpec('P')


class Invoker(Generic[P]):
    __slots__ = ('callback',)

    def __init__(self, callback: Callable[P, bool], /) -> None:
        self.callback: Callable[P, bool] = callback

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> bool:
        return self.callback(*args, **kwargs)


def inline(i: Invoker[int, int]) -> None:
    ...


def wrapped(i: Invoker[[int, int]]) -> None:
    ...


def with_tuple(i: Invoker[tuple[int, int]]) -> None:
    ...


print(get_type_hints(inline))
print(get_type_hints(wrapped))
print(get_type_hints(with_tuple))

And it outputs the following:

{'i': __main__.Invoker[(<class 'int'>, <class 'int'>)], 'return': <class 'NoneType'>}
{'i': __main__.Invoker[(<class 'int'>, <class 'int'>)], 'return': <class 'NoneType'>}
{'i': __main__.Invoker[(tuple[int, int],)], 'return': <class 'NoneType'>}

From this output, the syntax you're suggesting is wrong. Also the same program will crash in 3.9 and lower (ParamSpec has to be changed to be imported from typing_extensions) because of the errors I pointed out above. In a similar way to forward refs and union syntax, I assume there's a way to pass a string to the ParamSpec generic argument to prevent earlier versions of Python from crashing, but what should that string be? The PEP doesn't seem to outline that syntax or that type checkers should support the string syntax.

[srittau]

You are right, of course. For some reason I have thinking of TypeVar, not ParamSpec.

[bryanforbes]

I had to double check my code to make sure I was using a ParamSpec and not a TypeVar, so it’s not an unthinkable thing to do!! 😄

[JelleZijlstra]

Looks like we skipped this in typing-extensions tests: https://github.com/python/typing/blob/master/typing_extensions/src_py3/test_typing_extensions.py#L2075. I don't recall how hard we tried to get this to work, but perhaps it's possible with some more black magic in typing_extensions.

[bryanforbes]

I can understand skipping it because it would seem to require a change in Generic which may be impossible. I could see this being similar to the new union syntax in 3.9 where it can be used in strings (or in certain scenarios with from __future__ import annotations), but what is the syntax within that string? My first guess would be Invoker['[int, int]'], but I'd like to be sure of that before I start using it or suggesting it to type checker implementers.

[hauntsaninja]

The solutions in https://mypy.readthedocs.io/en/latest/runtime_troubles.html#using-classes-that-are-generic-in-stubs-but-not-at-runtime should be applicable here as well

[bryanforbes]

Yes, I’ve had to use that before, but I wasn’t sure if I would have to do that here if I can pass a string in the ParamSpec argument. I’m assuming I can use a string there (like with union syntax) but I’m not sure what the content of the string should be.