What's the correct way to check if an object is a typing.Generic?

Question

I'm trying to write code that validates type hints, and in order to do so I have to find out what kind of object the annotation is. For example, consider this snippet that's supposed to tell the user what kind of value is expected:

import typing

typ = typing.Union[int, str]

if issubclass(typ, typing.Union):
    print('value type should be one of', typ.__args__)
elif issubclass(typ, typing.Generic):
    print('value type should be a structure of', typ.__args__[0])
else:
    print('value type should be', typ)

This should print "value type should be one of (int, str)", but instead it throws an exception:

Traceback (most recent call last):
  File "untitled.py", line 6, in <module>
    if issubclass(typ, typing.Union):
  File "C:\Python34\lib\site-packages\typing.py", line 829, in __subclasscheck__
    raise TypeError("Unions cannot be used with issubclass().")
TypeError: Unions cannot be used with issubclass().

isinstance doesn't work either:

>>> isinstance(typ, typing.Union)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\site-packages\typing.py", line 826, in __instancecheck__
    raise TypeError("Unions cannot be used with isinstance().")
TypeError: Unions cannot be used with isinstance().

---

What's the correct way to check if typ is a typing.Generic?

If possible, I would like to see a solution that's backed by documentation or a PEP or some other resource. A "solution" that "works" by accessing undocumented, internal attributes is easy to find. But more likely than not, it'll turn out to be an implementation detail and will change in future versions. I'm looking for "the right way" to do it.

Solution 1

There is no official way to obtain this information. The typing module is still in heavy development, and has no public API to speak of. (In fact, it will probably never have one.)

All we can do is to look at the module's internals and find the least gross way to get the information we're after. And because the module is still being worked on, its internals will change. A lot.

---

In python 3.5 and 3.6, generics had an __origin__ attribute that held a reference to the original generic base class (i.e. List[int].__origin__ would've been List), but this was changed in 3.7. Now the easiest way to find out if something is a generic is probably to check its __parameters__ and __args__ attributes.

Here is a set of functions that can be used to detect generics:

import typing


if hasattr(typing, '_GenericAlias'):
    # python 3.7
    def _is_generic(cls):
        if isinstance(cls, typing._GenericAlias):
            return True

        if isinstance(cls, typing._SpecialForm):
            return cls not in {typing.Any}

        return False


    def _is_base_generic(cls):
        if isinstance(cls, typing._GenericAlias):
            if cls.__origin__ in {typing.Generic, typing._Protocol}:
                return False

            if isinstance(cls, typing._VariadicGenericAlias):
                return True

            return len(cls.__parameters__) > 0

        if isinstance(cls, typing._SpecialForm):
            return cls._name in {'ClassVar', 'Union', 'Optional'}

        return False
else:
    # python <3.7
    if hasattr(typing, '_Union'):
        # python 3.6
        def _is_generic(cls):
            if isinstance(cls, (typing.GenericMeta, typing._Union, typing._Optional, typing._ClassVar)):
                return True

            return False


        def _is_base_generic(cls):
            if isinstance(cls, (typing.GenericMeta, typing._Union)):
                return cls.__args__ in {None, ()}

            if isinstance(cls, typing._Optional):
                return True

            return False
    else:
        # python 3.5
        def _is_generic(cls):
            if isinstance(cls, (typing.GenericMeta, typing.UnionMeta, typing.OptionalMeta, typing.CallableMeta, typing.TupleMeta)):
                return True

            return False


        def _is_base_generic(cls):
            if isinstance(cls, typing.GenericMeta):
                return all(isinstance(arg, typing.TypeVar) for arg in cls.__parameters__)

            if isinstance(cls, typing.UnionMeta):
                return cls.__union_params__ is None

            if isinstance(cls, typing.TupleMeta):
                return cls.__tuple_params__ is None

            if isinstance(cls, typing.CallableMeta):
                return cls.__args__ is None

            if isinstance(cls, typing.OptionalMeta):
                return True

            return False


def is_generic(cls):
    """
    Detects any kind of generic, for example `List` or `List[int]`. This includes "special" types like
    Union and Tuple - anything that's subscriptable, basically.
    """
    return _is_generic(cls)


def is_base_generic(cls):
    """
    Detects generic base classes, for example `List` (but not `List[int]`)
    """
    return _is_base_generic(cls)


def is_qualified_generic(cls):
    """
    Detects generics with arguments, for example `List[int]` (but not `List`)
    """
    return is_generic(cls) and not is_base_generic(cls)

All of these functions should work in all python versions <= 3.7 (including anything <3.5 that uses the typing module backport).

Solution 2

You may be looking for __origin__:

# * __origin__ keeps a reference to a type that was subscripted,
#   e.g., Union[T, int].__origin__ == Union;`

import typing

typ = typing.Union[int, str]

if typ.__origin__ is typing.Union:
    print('value type should be one of', typ.__args__)
elif typ.__origin__ is typing.Generic:
    print('value type should be a structure of', typ.__args__[0])
else:
    print('value type should be', typ)

>>>value type should be one of (<class 'int'>, <class 'str'>)

The best I could find to advocate the use of this undocumented attribute is this reassuring quote from Guido Van Rossum (2 years ago):

The best I can recommend is using __origin__ -- if we were to change this attribute there would still have to be some other way to access the same information, and it would be easy to grep your code for occurrences of __origin__. (I'd be less worried about changes to __origin__ than to __extra__.) You may also look at the internal functions _gorg() and _geqv() (these names will not be part of any public API, obviously, but their implementations are very simple and conceptually useful).

This caveat in the documentation seem to indicate that nothing is set in marble yet:

New features might be added and API may change even between minor releases if deemed necessary by the core developers.

Solution 3

As pointed out by sonny-garcia in the comments, get_origin() works from python 3.8

import typing
from typing import get_origin

typ = typing.Union[int, str]
get_origin(typ) == typing.Union
#True

You can find more details in the docs

Comments

[Aran-Fey]

Looks promising. But it also looks like you found that by sifting through the source code. If at all possible, I'd prefer a solution that's backed by some documentation or a PEP or any other resource that indicates that this isn't just an implementation detail.

[Jacques Gaudin]

I haven't got any doc or PEP referencing it as typing is fairly recent. Is there anything about __args__ anywhere? I will be on the lookout.

[Aran-Fey]

I may be misunderstanding, but I don't see how get_type_hints would help me? If I define a variable x: typ and then use get_type_hints on it, I'll just get typing.Union[int, str] as the result.

[edixon]

Well, you have, at runtime, the 'int' and the 'str' you were looking for.

[edixon]

Or further inspect the result to craft a more detailed output, like you're code tries with isinstance and issubclass.

[Adam Williamson]

Sadly, this broke in Python 3.7. typing.Tuple[int, str].__origin__ is now the class tuple, not the class typing.Tuple. I have no great alternative yet :( You can do an awful string comparison that works, but... See bugzilla.redhat.com/show_bug.cgi?id=1598574 (this broke the Fedora / RHEL installer! Whee.)

[Sonny Garcia]

Python v3.8 adds the funcs typing.get_origin and typing.get_args. This option seems preferable to using their "magic" attribute counterparts.

[Aran-Fey]

@SonnyGarcia That's awesome, albeit 3 versions too late! Thanks for the heads up, I'll update my answer once I've had time to tinker with the new functions.

[alkasm]

@Aran-Fey ping on updating your answer :)

[Hernan]

What can be used for Python 3.8 as _VariadicGenericAlias is not there any more?

[Max Gasner]

There is a great compatibility layer for Python >=3.5 that backports typing.get_origin and typing.get_args: pypi.org/project/typing-compat. Be aware that the behavior of typing.get_args is still subtly different in 3.7 when called on the bare generics; in 3.8 typing.get_args(typing.Dict) is (), but in 3.7 it is (~KT, ~VT) (and analogously for the other generics), where ~KT and ~VT are objects of type typing.TypeVar.