From 61348b034e5941ae00d528cfc3bfad9fa6dcf45d Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 01/25] wip --- peps/pep-9999.rst | 163 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 peps/pep-9999.rst diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst new file mode 100644 index 00000000000..f298dab5320 --- /dev/null +++ b/peps/pep-9999.rst @@ -0,0 +1,163 @@ +PEP: 9999 +Title: AST Format for Annotation Functions +Author: Imogen Hergeth +Sponsor: Jelle Zijlstra +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 17-Dec-2025 +Python-Version: 3.15 +Post-History: `5-Nov-2025 ` + + +******** +Abstract +******** + +This PEP proposes that a new value, `Format.AST`, is added to the annotaionlib +enum and associated protocol. It instructs annotation functions to not evaluate +the annotation expressions directly, but rather to return the abstract syntax +trees that define each of them. This lets runtime consumers of annotations +observe their full definition, rather than just the object they evaluate to. +Which, in turn, creates the possibility of simplifying many existing type +annotations and using existing intuitive Python syntax in typing contexts. + +********** +Motivation +********** + +Background +========== + +When annotations were introduced to Python in :pep:`3107` and :pep:`526`, they +were defined simply as expressions that relate to a variable and are stored in +a new special dictionary ``__annotations__``. That is, writing +``var_name: expression`` just causes the Python interpreter to evaluate +``expression`` and store the result in the enclosing function's or class's +``__annotations__`` dictionary under the ``var_name`` key. + +This approach not only makes the implementation very simple, it also gives users +the freedom to use annotations to attach various kinds of metadata to variables. +One possibility are type hints, which specify what sets of values a variable is +expected to contain and how a function can safely be called. These have proven +to be very popular and have first received official support in :pep:`484` and +many additional features since. + +But this implementation also creates challanges for type hints. While simple +types such as the set of all integers can be spelled by just writing +``var: int``, there is no built-in syntax for more complicated typing concepts +like generics. When :pep:`484<484#generics>` introduced these, it thus had to +repurpose an existing form of expressions, namely indexing, and define the +needed operators on classes that should be usable in generic type annotations. +Note that while conceptually the expressions ``some_dict["key"]`` and +``list[int]`` denote very different operations, dictionary indexing and +specialisation of a generic type, Python treats them exactly the same, an +invocation of the ``__getitem__`` operator. + +:pep:`649` and :pep:`749` changed this behaviour in some ways. Instead of the +``__annotations__`` dictionary being built as the annotations are encountered, +the annotation's execution is now delayed until ``__annotations__`` is actually +accessed. Internally, a new method ``__annotate__`` is synthesised that creates +the dictionary by evaluation the class's or function's annotations. However, +what remains unchanged is that annotations are still treated as ordinary +expressions. They are evaluated just like any other expression in a different +context, their execution just is delayed until they are needed. + +Problems with the Current Approach +================================== + +This behaviour can cause problems where users want to use an expression in +annotations in a way that clashes with the expressions usual behaviour. For +example, consider literal types, i.e. types that consist of some particular set +of literal values. Currently, these are written as e.g. ``Literal[1]``. Many +users of type annotations would prefer to drop the redundant ``Literal[]``, the +context of it occurring in a tpye annotation already makes it clear that the +``1`` represents the literal type containing only ``1``. This also is reflected +in other languages, such as typescript, implementing literal types that way. + +The problem arises when these literal types are combined with other typing +constructs. For example, the union type ``1 | 2 | 3``, representing values that +are either ``1``, ``2`` or ``3``, cannot be properly evaluated at runtime. + + + Other +languages such as typescript write the same type simply as `1 | 2 | 3`, omitting +the redundant `Literal` container. This greatly simplifies more complex types +such as multidimensional arrays with known sizes like +`ndarray[tuple[Literal[16], Literal[1000], Literal[1000]], dtype[uint8]]`. +This cannot currently be done in Python since the `|` operator already is +defined for `int` as bitwise or. That means that a type annotation like +`1 | 2 | 3` is evaluated to the object `3` with no way of recovering the +annotation's union information at runtime. We are forced to add the `Literal` +wrapper to make sure that the objects that are being created carry all necessary +metadata and implement all operations correctly. + + + +annotations define ``some_type | other_type`` to denote the +union of the two types. To support this, every legal type expression needs to +resolve to an object that implements ``__or__`` in such a way that it returns +the appropriate ``typing.UnionType`` object. + + + + +The problem occurrs in cases where such a workaround is not possible. Consider, +for example, the case of literal types. That is, types that contain some +specific set of literal values, currently written as `Literal[1, 2, 3]`. Other +languages such as typescript write the same type simply as `1 | 2 | 3`, omitting +the redundant `Literal` container. This greatly simplifies more complex types +such as multidimensional arrays with known sizes like +`ndarray[tuple[Literal[16], Literal[1000], Literal[1000]], dtype[uint8]]`. +This cannot currently be done in Python since the `|` operator already is +defined for `int` as bitwise or. That means that a type annotation like +`1 | 2 | 3` is evaluated to the object `3` with no way of recovering the +annotation's union information at runtime. We are forced to add the `Literal` +wrapper to make sure that the objects that are being created carry all necessary +metadata and implement all operations correctly. + +The builtin container types also cannot be spelled as `(int, str)`, `[int]` or +`{int: str}` but need to use the generic syntax like `tuple[int, str]`, +`list[int]` or `dict[int, str]`. This again is because tuple, list and dict +objects have already defined runtime behaviour that does not work within the +context of type annotations. + +While all of these examples do have existing workarounds, the +additional work required to spell these types and their readability issues do +present real problems. Many Python users, particulary those new to typing, do +not intuitively understand what a construct like +`tuple[Literal[16], Literal[1000]]` is supposed to mean and what it is doing in +an array annotation. Being able to simply write the array shape as `(16, 1000)` +makes it much clearer what is meant. + +These examples also show that the annotations' specification of being treated +as any other expression is not being honored very well by this implementation. +When a user creates an array object they can specify its shape by just writing +`ndarray(..., shape=(16, 1000))`. But even though the type annotation is +supposed to caputre the exact same information, and is meant to "just be an +expression", they are forced to write `tuple[Literal[16], Litearl[1000]]` +instead. + +There also are new typing features that are being held back by this requirement +to create syntactical workarounds. For example, inline typed dictionaries could +intuitively be defined inline as `{"some_key": int, "other_key": str}`. But +this again does not work because the semantics of `dict` objects to not work +properly in typing contexts. There is `previous discussion +` +of this, with the main hurdle to implementation being the runtime behaviour. +Other examples are `conditional types +`, which +let users write ternary statements in type definitions, like +`type SomeAlias[T] = int if issubclass(T, str) else str`. Or extending integer +literal types to support basic arithmetic operations. This would enable array +libraries to properly track shapes across operations like concatenation. The +existing `work on this +` +unfortunately concluded that while this is a very useful feature for many users, +implementing it with current semantics is too verbose and cumbersome to gain +much traction. + +Scope +===== + + From 43fe5fbb13a8221e354115c05bc4115efb968022 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 02/25] finish motivation --- peps/pep-9999.rst | 102 ++++++++++++++++------------------------------ 1 file changed, 36 insertions(+), 66 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index f298dab5320..c8c8a65ceba 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -77,87 +77,57 @@ in other languages, such as typescript, implementing literal types that way. The problem arises when these literal types are combined with other typing constructs. For example, the union type ``1 | 2 | 3``, representing values that -are either ``1``, ``2`` or ``3``, cannot be properly evaluated at runtime. - - - Other -languages such as typescript write the same type simply as `1 | 2 | 3`, omitting -the redundant `Literal` container. This greatly simplifies more complex types -such as multidimensional arrays with known sizes like -`ndarray[tuple[Literal[16], Literal[1000], Literal[1000]], dtype[uint8]]`. -This cannot currently be done in Python since the `|` operator already is -defined for `int` as bitwise or. That means that a type annotation like -`1 | 2 | 3` is evaluated to the object `3` with no way of recovering the -annotation's union information at runtime. We are forced to add the `Literal` -wrapper to make sure that the objects that are being created carry all necessary -metadata and implement all operations correctly. - - - -annotations define ``some_type | other_type`` to denote the -union of the two types. To support this, every legal type expression needs to -resolve to an object that implements ``__or__`` in such a way that it returns -the appropriate ``typing.UnionType`` object. - - - - -The problem occurrs in cases where such a workaround is not possible. Consider, -for example, the case of literal types. That is, types that contain some -specific set of literal values, currently written as `Literal[1, 2, 3]`. Other -languages such as typescript write the same type simply as `1 | 2 | 3`, omitting -the redundant `Literal` container. This greatly simplifies more complex types -such as multidimensional arrays with known sizes like -`ndarray[tuple[Literal[16], Literal[1000], Literal[1000]], dtype[uint8]]`. -This cannot currently be done in Python since the `|` operator already is -defined for `int` as bitwise or. That means that a type annotation like -`1 | 2 | 3` is evaluated to the object `3` with no way of recovering the -annotation's union information at runtime. We are forced to add the `Literal` -wrapper to make sure that the objects that are being created carry all necessary -metadata and implement all operations correctly. - -The builtin container types also cannot be spelled as `(int, str)`, `[int]` or -`{int: str}` but need to use the generic syntax like `tuple[int, str]`, -`list[int]` or `dict[int, str]`. This again is because tuple, list and dict -objects have already defined runtime behaviour that does not work within the -context of type annotations. +are either ``1``, ``2`` or ``3``, cannot be properly evaluated at runtime. We +would want it to evaluate to a ``UnionType`` containing references to +``1, 2, 3``. But since the ``__or__`` special method is already implemented on +``int`` as the bitwise or operation, it will just be evaluated to ``3``. There +is no way of recovering the ``1`` and ``2`` present in the annotation from the +``__annotation__`` dictionary. + +Similar issues prevent us from using display syntax to denote the built-in +container types, forcing us to write ``set[int]``, ``list[int]`` and +``dict[int, str]`` instead of just ``{int}``, ``[int]`` and ``{int: str}``. +This is because these the display syntax causes e.g. a ``set`` object to be +constructed, rather than a special object representing a type. And ``set`` +already implements ``__or__`` to create a new set containing all elements of +both arguments, rather than a ``Union`` object. While all of these examples do have existing workarounds, the additional work required to spell these types and their readability issues do -present real problems. Many Python users, particulary those new to typing, do -not intuitively understand what a construct like -`tuple[Literal[16], Literal[1000]]` is supposed to mean and what it is doing in -an array annotation. Being able to simply write the array shape as `(16, 1000)` -makes it much clearer what is meant. - -These examples also show that the annotations' specification of being treated -as any other expression is not being honored very well by this implementation. -When a user creates an array object they can specify its shape by just writing -`ndarray(..., shape=(16, 1000))`. But even though the type annotation is -supposed to caputre the exact same information, and is meant to "just be an -expression", they are forced to write `tuple[Literal[16], Litearl[1000]]` -instead. +present real problems. Many Python users, particulary those new to typing, +intuitively reach towards the easier and shorter syntax like ``(int, str)`` +to denote a tuple type. The more verbose syntax also is cumbersome to understand +when it occurrs as a type argument for a generic type. A very common example are +matrix libraries like numpy that use integer tuples to define a matrix's shape. +When creating such a matrix, you simply write +``ndarray(..., shape=(16, 1000))``. But that matrix's type is spelled as +``ndarray[tuple[Literal[16], Literal[1000]], ...]``. A user unfamiliar with +the internals of the Python type system is hard-pressed to see such an +annotation and understand what information it is trying to tell them and why +they have to use these seemingly redundant ``Literal`` tags. + +This example also shows that the annotation specification's intention of them +being treated as any other expression is not being honored very well by this +implementation. While an annotation can contain arbitrary expressions, this is +only true from the interpreter's point of view. The vast majority of users, +which use type annotations have to switch from the familiar ``(16, 1000)`` +syntax to the annotation-specific ``tuple[Literal[16], Literal[1000]]``. There also are new typing features that are being held back by this requirement to create syntactical workarounds. For example, inline typed dictionaries could intuitively be defined inline as `{"some_key": int, "other_key": str}`. But this again does not work because the semantics of `dict` objects to not work properly in typing contexts. There is `previous discussion -` +`__ of this, with the main hurdle to implementation being the runtime behaviour. Other examples are `conditional types -`, which -let users write ternary statements in type definitions, like +`__, +which let users write ternary statements in type definitions, like `type SomeAlias[T] = int if issubclass(T, str) else str`. Or extending integer literal types to support basic arithmetic operations. This would enable array libraries to properly track shapes across operations like concatenation. The existing `work on this -` +`__ unfortunately concluded that while this is a very useful feature for many users, implementing it with current semantics is too verbose and cumbersome to gain much traction. - -Scope -===== - - From 9a39fd9f1aa5efbf0cf7c8d3a7d251d5068270f9 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 03/25] headings --- peps/pep-9999.rst | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index c8c8a65ceba..8835fde4814 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -131,3 +131,35 @@ existing `work on this unfortunately concluded that while this is a very useful feature for many users, implementing it with current semantics is too verbose and cumbersome to gain much traction. + +********* +Rationale +********* + +************* +Specification +************* + +*********************** +Backwards Compatibility +*********************** + +********************* +Security Implications +********************* + +***************** +How to Teach This +***************** + +************************ +Reference Implementation +************************ + +************** +Rejected Ideas +************** + +******** +Copyright +********* From a3914ca4910f1ad6b83eb3ff0632a44c31f36ad2 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 04/25] copyright --- peps/pep-9999.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 8835fde4814..af478996dc2 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -163,3 +163,6 @@ Rejected Ideas ******** Copyright ********* + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 2c81c844db287bc2ec960778349fbdb110ba3a67 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 05/25] rationale --- peps/pep-9999.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index af478996dc2..44f17dd8ba0 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -136,6 +136,21 @@ much traction. Rationale ********* +In this PEP, we adress the above issues by proposing to expand the functionality +of ``__annotate__`` methods to support a new format, ``AST``, which instructs +them to return the abstract syntax tree of the class's or function's +annotations, rather than their value. This enables users of annotations to +inspect the annotation expression and evaluate it using domain specific +semantics that might differ from simply evaluating a Python expression. This +does not only benefit people using annotations for type hints, but also supports +other uses of annotations. + +Note that we do not propose adding direct support for any of the specific +type annotation changes or additions that are discussed in this PEP and restrict +this proposal to laying the necessary groundwork that enables such changes. +These specific changes are mentioned only as motivating examples to show why +the ``AST`` format is needed. + ************* Specification ************* From cab07bc82e123501fd27aed7b422c7653af53427 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 06/25] cleanup --- peps/pep-9999.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 44f17dd8ba0..f4e1c539534 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -14,13 +14,14 @@ Post-History: `5-Nov-2025 Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 07/25] specification --- peps/pep-9999.rst | 94 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index f4e1c539534..a828e1f45bc 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -156,6 +156,100 @@ format is needed. Specification ************* +Throughout this PEP, when we talk about *annotation functions/methods* we are +referring to both the ``__annotate__`` special methods found on some objects +and the following methods found on typing objects: + +* ``evaluate_value`` on :py:class:`typing.TypeAliasType` +* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on + :py:class:`typing.TypeVar` +* ``evaluate_default`` on :py:class:`typing.ParamSpec` +* ``evaluate_default`` on :py:class:`typing.TypeVarTuple` + +When referring to their return values, we mean either the objects contained in +the dictionary returned by ``__annotate__`` methods or the single object +returned by the other methods. + +The ``AST`` Format +================== + +A new value called ``AST`` is added to the ``annotationlib.Format`` enum with +value 5. Annotation functions do not have to support this format. If an +annotation function is called with this format, it must return instances of +``ast.expr`` that do not contain ``ast.NamedExpr``, ``ast.Yield``, +``ast.YieldFrom`` or ``ast.Await`` nodes (the Python grammar +already prohibits the use of these features in annotations). + +Compiler-generated annotation functions will always support this format +and return the abstract syntax tree of the class's or function's annotations. +They will store the necessary AST data stored as tuple constants and then +construct a new ``ast.expr`` object each time they are called. These objects +will be functionally identical to the objects created by parsing the annotation +code directly. + +Annotate functions that are generated by library code will need to manually +support the new format or choose to raise ``NotImplementedError()`` when it +is requrested. Most of these use cases synthesise new annotation functions by +introspecting existing function or class objects and combining and/or modifying +their return values. The new format makes this easier to do since the AST +objects can always be created regardless of the existence of forward references +or similar issues, they can be modified to denote the synthesised annotations +using existing tools and they can be evaluated to create the objects requested +by other formats. + +Helper Functions +================ + +The existing ``typing.get_type_hints`` function will be modified to accept a +new, optional Boolean keyword argument ``use_type_syntax`` that defaults to +``False``. When set to true, ``get_type_hints`` does not call the underlying +annotation function with the requested format, but always with ``Format.AST``. +The returned AST is then evaluated to an object, using the semantics of Python +expressions, modified by rules that are defined in the typing spec. At the time +this PEP is written, no such rules exist and thus the AST is always evaluated +normally. + +When the keyword argument is set, any annotation expressions that are not +valid type annotation expressions, as `defined by the typing spec +__`, +cause ``get_type_hints`` to raise a ``SyntaxError()``. + +When the typing spec is changed to define new type expressions, +``get_type_hints`` will be modified accordingly and return a particular object +when it previously would have raised a ``SyntaxError`` on the same inputs. This +is an exemption to the usual deprecation policy and will be documented as such. +Other behaviour of ``get_type_hints`` will not be affected, users can still rely +on the stability of returned values and other raised exceptions. + +In order to ease the creation of synthesised annotation functions, a new helper +function ``create_annotate_from_asts(annotations: dict[str, tuple[ast.expr, +Mapping[str, object]]]) -> Callable[[Format], dict[str, object]]`` will be +added to +:py:mod:`annotationlib`. It creates an ``__annotate__`` method that supports +the ``VALUE``, ``FORWARDREF``, ``SOURCE`` and ``AST`` formats. The returned +values are the objects of the correct type that would be created if each of +the passed syntax trees would be evaluated in the corresponding namespace. + +Pseudocode +========== +A function defined as ``def f(first: int, second: tuple[int, str])`` will have +a compiler-generated ``__annotate__`` method that might look something like this +were it written in Python:: + + def __annotate__(self, format): + if format <= 2: + return { + "first": int, + "second": tuple[int, str], + } + if format == 5: + from ast import _ast_from_tuple + return { + "first": _ast_from_tuple((26, "int", 1, 1, 1, 14)), + "second": _ast_from_tuple((...)), + } + raise NotImplementedError + *********************** Backwards Compatibility *********************** From be919dbaca9d4d7f96ae4b3f100498d9a99eed82 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 08/25] backwards compatibility --- peps/pep-9999.rst | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index a828e1f45bc..06db5e23d5b 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -23,6 +23,8 @@ object they evaluate to. Which, in turn, creates the possibility of simplifying many existing type annotations and using existing intuitive Python syntax in typing contexts. +.. _motivation + ********** Motivation ********** @@ -254,6 +256,37 @@ were it written in Python:: Backwards Compatibility *********************** +Since this PEP only adds new functionality, there are no direct backwards +compatibility concerns. However, we also want to point out that future additions +like the ones outlined in the :ref:`motivation` do not create backwards +compatibility problems, even if adopted gradually. + +Consider, for example, a change to the typing spec that made it so that +``var: 1`` is interpreted as a literal type ``Literal[1]``. When a user +evaluates the annotation method directly or with one of the helper methods, they +will still observe the result as the plain integer ``1``. The changed semantics +only happen when the user opts in to the new semantics by calling +``typing.get_type_hints`` with the new keyword argument set. + +In the most common use case, annotations are not consumed by the user directly +but by some library code that introspects user defined objects. Thus, adoption +of new annotation semantics could be stalled if library authors have to worry +about existing user annotations being re-interpreted to different objects when +the library is updated. But this also is not an issue. The typing spec already +is covered by backwards compatibility concerns, which means that any currently +valid type annotations will not be changed to mean something different. +Potential future changes can only define new semantics to syntax constructs that +currently are not valid type annotations and thus do not occur in user code. + +The only potential for issues to happen is if a user has written annotations +that currently are not legal type annotations, but become legal with different +semantics in the future. We believe that this case is exceedingly rare since +such users either do not intend to use annotations for type hints and will thus +not use a library that expects them to be type hints, or the code will already +not work since the annotations currently cannot be handled correctly by the +library code. + + ********************* Security Implications ********************* From 91a2576342e47eb48f93aece0f9c2b60c850a41f Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 09/25] security, teach, reference impl --- peps/pep-9999.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 06db5e23d5b..3e116e8174d 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -291,14 +291,33 @@ library code. Security Implications ********************* +There are no known security implications for this change. Calling annotation +functions already could execute arbitrary code defined in the annotated +object. Python also already offers the capability to access the source code and +compiled byte code of introspected objects. The AST objects returned in the +new format thus do not contain any previously unavailable information. + ***************** How to Teach This ***************** +Users of annotations are unaffected by this proposal. It intends to enable the +simplification of the way type annotations are spelled, any future changes +based on this PEP will need to be evaluated on their own merits. + +The new format and changes to helper functions will be documented as part of the +language standard. Libraries that introspect type annotations will be able to +easily support any new type syntax by calling ``typing.get_type_hints`` and +should document this behaviour so that their users are made aware of any +potential future changes. + ************************ Reference Implementation ************************ +This proposal is prototyped in `a CPython fork +`__. + ************** Rejected Ideas ************** From d86c855c26deb5a6a7aa5fa2634bf3a95dbbc0f8 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 10/25] backwards compatibility --- peps/pep-9999.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 3e116e8174d..a6aea61b845 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -322,6 +322,17 @@ This proposal is prototyped in `a CPython fork Rejected Ideas ************** +A New Argument for Annotation Functions +======================================= + +An initial idea was to add a new optional argument to annotation functions, +which instructs them to return objects of the specified format's type, but +using a type annotation specific syntax. This was rejected since it means that +any type-specific syntax needs to be defined in the interpreter itself, which +limits the kinds of semantics that are possible. It also locks the usage of +typing features to the Python version that is being used, rather than allowing +the usual backporting via :py:mod:`typing_extensions`. + ******** Copyright ********* From a2ffa72abe83dec780583862ab72935e220c2c1b Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 11/25] eval_type --- peps/pep-9999.rst | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index a6aea61b845..617ac74415c 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -154,6 +154,17 @@ this proposal to laying the necessary groundwork that enables them. Such changes are mentioned only as motivating examples to show why the ``AST`` format is needed. +In order to support such future additions to the typing spec, the +``typing.get_type_hints`` helper function will support a mode that uses the +``AST`` format to evaluate type annotations with the typing spec's semantics. + +There are some places other than annotations where users need to spell a type, +for example the first argument to ``typing.cast``. These use cases will be +covered by the existing practice of wrapping types in string literals. If such +a type needs to be introspected at runtime, a new helper function +``typing.eval_type`` can be used, which internally uses the new format to +evaluate a string under the typing spec's semantics. + ************* Specification ************* @@ -223,6 +234,13 @@ is an exemption to the usual deprecation policy and will be documented as such. Other behaviour of ``get_type_hints`` will not be affected, users can still rely on the stability of returned values and other raised exceptions. +A new helper method ``eval_type(tp: str, globals: Mapping[str, object], +locals: Mapping[str, object], format: Format = Format.VALUE) -> TypeForm`` +will be added to the :py:mod:`typing` module. It evaluates a string using the +same rules that ``get_type_hints`` uses with the type-specific semantics. +This function can be used to evaluate stringified types that are used in places +other than annotations. + In order to ease the creation of synthesised annotation functions, a new helper function ``create_annotate_from_asts(annotations: dict[str, tuple[ast.expr, Mapping[str, object]]]) -> Callable[[Format], dict[str, object]]`` will be @@ -301,9 +319,16 @@ new format thus do not contain any previously unavailable information. How to Teach This ***************** -Users of annotations are unaffected by this proposal. It intends to enable the -simplification of the way type annotations are spelled, any future changes -based on this PEP will need to be evaluated on their own merits. +Users of annotations are not directly affected by this proposal. It intends to +enable the simplification of the way type annotations are spelled, any future +changes based on this PEP will need to be evaluated on their own merits. +Documentation will inform users that any new semantics is only natively +supported in annotations. Since this is by far the most common place for types +to be spelled, we expect this to not be a big limitation. Other places where +types can occur already require users to wrap forward references in string +literals, so this is a known practice. Type checkers should warn users if they +do not wrap a type form using syntax that would be evaluated incorrectly in +a place where it is statically known that a type form is expected. The new format and changes to helper functions will be documented as part of the language standard. Libraries that introspect type annotations will be able to From f032d77053f6fed3f4db930aef46b9fe5a7ba1fd Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 12/25] linting --- peps/pep-9999.rst | 104 +++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 52 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 617ac74415c..a5c39abf4cd 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -4,17 +4,18 @@ Author: Imogen Hergeth Sponsor: Jelle Zijlstra Status: Draft Type: Standards Track -Content-Type: text/x-rst +Topic: Typing +Requires: 749 Created: 17-Dec-2025 Python-Version: 3.15 -Post-History: `5-Nov-2025 ` +Post-History: `05-Nov-2025 `__ ******** Abstract ******** -This PEP proposes that a new value, `Format.AST`, is added to the +This PEP proposes that a new value, ``Format.AST``, is added to the :py:mod:`annotaionlib` enum and associated protocol. It instructs annotation functions to not evaluate the annotation expressions directly, but rather to return the abstract syntax trees that define each of them. This lets runtime @@ -23,7 +24,7 @@ object they evaluate to. Which, in turn, creates the possibility of simplifying many existing type annotations and using existing intuitive Python syntax in typing contexts. -.. _motivation +.. _motivation: ********** Motivation @@ -46,7 +47,7 @@ expected to contain and how a function can safely be called. These have proven to be very popular and have first received official support in :pep:`484` and many additional features since. -But this implementation also creates challanges for type hints. While simple +But this implementation also creates challenges for type hints. While simple types such as the set of all integers can be spelled by just writing ``var: int``, there is no built-in syntax for more complicated typing concepts like generics. When :pep:`484<484#generics>` introduced these, it thus had to @@ -54,14 +55,14 @@ repurpose an existing form of expressions, namely indexing, and define the needed operators on classes that should be usable in generic type annotations. Note that while conceptually the expressions ``some_dict["key"]`` and ``list[int]`` denote very different operations, dictionary indexing and -specialisation of a generic type, Python treats them exactly the same, an +specialization of a generic type, Python treats them exactly the same, an invocation of the ``__getitem__`` operator. -:pep:`649` and :pep:`749` changed this behaviour in some ways. Instead of the +:pep:`649` and :pep:`749` changed this behavior in some ways. Instead of the ``__annotations__`` dictionary being built as the annotations are encountered, the annotation's execution is now delayed until ``__annotations__`` is actually -accessed. Internally, a new method ``__annotate__`` is synthesised that creates -the dictionary by evaluation the class's or function's annotations. However, +accessed. Internally, a new method ``__annotate__`` is synthesized that creates +the dictionary by evaluating the class's or function's annotations. However, what remains unchanged is that annotations are still treated as ordinary expressions. They are evaluated just like any other expression in a different context, their execution just is delayed until they are needed. @@ -69,14 +70,14 @@ context, their execution just is delayed until they are needed. Problems with the Current Approach ================================== -This behaviour can cause problems where users want to use an expression in -annotations in a way that clashes with the expressions usual behaviour. For +This behavior can cause problems where users want to use an expression in +annotations in a way that clashes with the expression's usual behavior. For example, consider literal types, i.e. types that consist of some particular set of literal values. Currently, these are written as e.g. ``Literal[1]``. Many users of type annotations would prefer to drop the redundant ``Literal[]``, the -context of it occurring in a tpye annotation already makes it clear that the +context of it occurring in a type annotation already makes it clear that the ``1`` represents the literal type containing only ``1``. This also is reflected -in other languages, such as typescript, implementing literal types that way. +in other languages, such as TypeScript, implementing literal types that way. The problem arises when these literal types are combined with other typing constructs. For example, the union type ``1 | 2 | 3``, representing values that @@ -90,17 +91,17 @@ is no way of recovering the ``1`` and ``2`` present in the annotation from the Similar issues prevent us from using display syntax to denote the built-in container types, forcing us to write ``set[int]``, ``list[int]`` and ``dict[int, str]`` instead of just ``{int}``, ``[int]`` and ``{int: str}``. -This is because these the display syntax causes e.g. a ``set`` object to be -constructed, rather than a special object representing a type. And ``set`` +For example, in the first case the display syntax causes a ``set`` object to be +constructed rather than a special object representing a type. And ``set`` already implements ``__or__`` to create a new set containing all elements of both arguments, rather than a ``Union`` object. While all of these examples do have existing workarounds, the additional work required to spell these types and their readability issues do -present real problems. Many Python users, particulary those new to typing, +present real problems. Many Python users, particularly those new to typing, intuitively reach towards the easier and shorter syntax like ``(int, str)`` to denote a tuple type. The more verbose syntax also is cumbersome to understand -when it occurrs as a type argument for a generic type. A very common example are +when it occurs as a type argument for a generic type. A very common example are matrix libraries like numpy that use integer tuples to define a matrix's shape. When creating such a matrix, you simply write ``ndarray(..., shape=(16, 1000))``. But that matrix's type is spelled as @@ -118,15 +119,15 @@ syntax to the annotation-specific ``tuple[Literal[16], Literal[1000]]``. There also are new typing features that are being held back by this requirement to create syntactical workarounds. For example, inline typed dictionaries could -intuitively be defined inline as `{"some_key": int, "other_key": str}`. But -this again does not work because the semantics of `dict` objects to not work +intuitively be defined inline as ``{"some_key": int, "other_key": str}``. But +this again does not work because the semantics of ``dict`` objects to not work properly in typing contexts. There is `previous discussion `__ -of this, with the main hurdle to implementation being the runtime behaviour. +of this, with the main hurdle to implementation being the runtime behavior. Other examples are `conditional types `__, which let users write ternary statements in type definitions, like -`type SomeAlias[T] = int if issubclass(T, str) else str`. Or extending integer +``type SomeAlias[T] = int if issubclass(T, str) else str``. Or extending integer literal types to support basic arithmetic operations. This would enable array libraries to properly track shapes across operations like concatenation. The existing `work on this @@ -139,14 +140,14 @@ much traction. Rationale ********* -In this PEP, we adress the above issues by proposing to expand the functionality -of ``__annotate__`` methods to support a new format, ``AST``, which instructs -them to return the abstract syntax tree of the class's or function's -annotations, rather than their value. This enables users of annotations to -inspect the annotation expression and evaluate it using domain specific -semantics that might differ from simply evaluating a Python expression. This -does not only benefit people using annotations for type hints, but also supports -other uses of annotations. +In this PEP, we address the above issues by proposing to expand the +functionality of ``__annotate__`` methods to support a new format, ``AST``, +which instructs them to return the abstract syntax tree of the class's or +function's annotations, rather than their value. This enables users of +annotations to inspect the annotation expression and evaluate it using +domain-specific semantics that might differ from simply evaluating a Python +expression. This does not only benefit people using annotations for type +hints but also supports other uses of annotations. Note that we do not propose adding direct support for any of the specific type annotation changes or additions that are discussed in this PEP and restrict @@ -159,7 +160,7 @@ In order to support such future additions to the typing spec, the ``AST`` format to evaluate type annotations with the typing spec's semantics. There are some places other than annotations where users need to spell a type, -for example the first argument to ``typing.cast``. These use cases will be +for example, the first argument to ``typing.cast``. These use cases will be covered by the existing practice of wrapping types in string literals. If such a type needs to be introspected at runtime, a new helper function ``typing.eval_type`` can be used, which internally uses the new format to @@ -169,13 +170,12 @@ evaluate a string under the typing spec's semantics. Specification ************* -Throughout this PEP, when we talk about *annotation functions/methods* we are +Throughout this PEP, when we talk about *annotation functions/methods*, we are referring to both the ``__annotate__`` special methods found on some objects and the following methods found on typing objects: * ``evaluate_value`` on :py:class:`typing.TypeAliasType` -* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on - :py:class:`typing.TypeVar` +* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on :py:class:`typing.TypeVar` * ``evaluate_default`` on :py:class:`typing.ParamSpec` * ``evaluate_default`` on :py:class:`typing.TypeVarTuple` @@ -202,12 +202,12 @@ code directly. Annotate functions that are generated by library code will need to manually support the new format or choose to raise ``NotImplementedError()`` when it -is requrested. Most of these use cases synthesise new annotation functions by +is requested. Most of these use cases synthesize new annotation functions by introspecting existing function or class objects and combining and/or modifying their return values. The new format makes this easier to do since the AST objects can always be created regardless of the existence of forward references -or similar issues, they can be modified to denote the synthesised annotations -using existing tools and they can be evaluated to create the objects requested +or similar issues, they can be modified to denote the synthesized annotations +using existing tools, and they can be evaluated to create the objects requested by other formats. Helper Functions @@ -216,10 +216,10 @@ Helper Functions The existing ``typing.get_type_hints`` function will be modified to accept a new, optional Boolean keyword argument ``use_type_syntax`` that defaults to ``False``. When set to true, ``get_type_hints`` does not call the underlying -annotation function with the requested format, but always with ``Format.AST``. +annotation function with the requested format but always with ``Format.AST``. The returned AST is then evaluated to an object, using the semantics of Python expressions, modified by rules that are defined in the typing spec. At the time -this PEP is written, no such rules exist and thus the AST is always evaluated +this PEP is written, no such rules exist, and thus the AST is always evaluated normally. When the keyword argument is set, any annotation expressions that are not @@ -231,7 +231,7 @@ When the typing spec is changed to define new type expressions, ``get_type_hints`` will be modified accordingly and return a particular object when it previously would have raised a ``SyntaxError`` on the same inputs. This is an exemption to the usual deprecation policy and will be documented as such. -Other behaviour of ``get_type_hints`` will not be affected, users can still rely +Other behavior of ``get_type_hints`` will not be affected; users can still rely on the stability of returned values and other raised exceptions. A new helper method ``eval_type(tp: str, globals: Mapping[str, object], @@ -241,14 +241,14 @@ same rules that ``get_type_hints`` uses with the type-specific semantics. This function can be used to evaluate stringified types that are used in places other than annotations. -In order to ease the creation of synthesised annotation functions, a new helper +In order to ease the creation of synthesized annotation functions, a new helper function ``create_annotate_from_asts(annotations: dict[str, tuple[ast.expr, Mapping[str, object]]]) -> Callable[[Format], dict[str, object]]`` will be added to -:py:mod:`annotationlib`. It creates an ``__annotate__`` method that supports +``annotationlib``. It creates an ``__annotate__`` method that supports the ``VALUE``, ``FORWARDREF``, ``SOURCE`` and ``AST`` formats. The returned values are the objects of the correct type that would be created if each of -the passed syntax trees would be evaluated in the corresponding namespace. +the passed syntax trees were evaluated in the corresponding namespace. Pseudocode ========== @@ -287,9 +287,9 @@ only happen when the user opts in to the new semantics by calling ``typing.get_type_hints`` with the new keyword argument set. In the most common use case, annotations are not consumed by the user directly -but by some library code that introspects user defined objects. Thus, adoption +but by some library code that introspects user-defined objects. Thus, adoption of new annotation semantics could be stalled if library authors have to worry -about existing user annotations being re-interpreted to different objects when +about existing user annotations being reinterpreted to different objects when the library is updated. But this also is not an issue. The typing spec already is covered by backwards compatibility concerns, which means that any currently valid type annotations will not be changed to mean something different. @@ -297,7 +297,7 @@ Potential future changes can only define new semantics to syntax constructs that currently are not valid type annotations and thus do not occur in user code. The only potential for issues to happen is if a user has written annotations -that currently are not legal type annotations, but become legal with different +that currently are not legal type annotations but become legal with modified semantics in the future. We believe that this case is exceedingly rare since such users either do not intend to use annotations for type hints and will thus not use a library that expects them to be type hints, or the code will already @@ -320,7 +320,7 @@ How to Teach This ***************** Users of annotations are not directly affected by this proposal. It intends to -enable the simplification of the way type annotations are spelled, any future +enable the simplification of the way type annotations are spelled; any future changes based on this PEP will need to be evaluated on their own merits. Documentation will inform users that any new semantics is only natively supported in annotations. Since this is by far the most common place for types @@ -333,7 +333,7 @@ a place where it is statically known that a type form is expected. The new format and changes to helper functions will be documented as part of the language standard. Libraries that introspect type annotations will be able to easily support any new type syntax by calling ``typing.get_type_hints`` and -should document this behaviour so that their users are made aware of any +should document this behavior so that their users are made aware of any potential future changes. ************************ @@ -351,14 +351,14 @@ A New Argument for Annotation Functions ======================================= An initial idea was to add a new optional argument to annotation functions, -which instructs them to return objects of the specified format's type, but -using a type annotation specific syntax. This was rejected since it means that -any type-specific syntax needs to be defined in the interpreter itself, which +which instructs them to return objects of the specified format's type but +using type annotation-specific semantics. This was rejected since it means that +the type-specific semantics need to be defined in the interpreter itself, which limits the kinds of semantics that are possible. It also locks the usage of typing features to the Python version that is being used, rather than allowing -the usual backporting via :py:mod:`typing_extensions`. +the usual backporting via ``typing_extensions``. -******** +********* Copyright ********* From 2a1bc9d0d3d22d8a7ee447f2937be6f8ce03534c Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:14 +0200 Subject: [PATCH 13/25] fix reference --- peps/pep-9999.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index a5c39abf4cd..254c52a6cd6 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -16,7 +16,7 @@ Abstract ******** This PEP proposes that a new value, ``Format.AST``, is added to the -:py:mod:`annotaionlib` enum and associated protocol. It instructs annotation +:py:mod:`annotationlib` enum and associated protocol. It instructs annotation functions to not evaluate the annotation expressions directly, but rather to return the abstract syntax trees that define each of them. This lets runtime consumers of annotations observe their full definition, rather than just the @@ -245,7 +245,7 @@ In order to ease the creation of synthesized annotation functions, a new helper function ``create_annotate_from_asts(annotations: dict[str, tuple[ast.expr, Mapping[str, object]]]) -> Callable[[Format], dict[str, object]]`` will be added to -``annotationlib``. It creates an ``__annotate__`` method that supports +:py:mod:`annotationlib`. It creates an ``__annotate__`` method that supports the ``VALUE``, ``FORWARDREF``, ``SOURCE`` and ``AST`` formats. The returned values are the objects of the correct type that would be created if each of the passed syntax trees were evaluated in the corresponding namespace. From 7c6ae2d15bfd344acbba93802d74764595b03ce9 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:15 +0200 Subject: [PATCH 14/25] wip --- peps/pep-9999.rst | 124 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 110 insertions(+), 14 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 254c52a6cd6..9c8e24e4d1f 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -140,20 +140,116 @@ much traction. Rationale ********* -In this PEP, we address the above issues by proposing to expand the -functionality of ``__annotate__`` methods to support a new format, ``AST``, -which instructs them to return the abstract syntax tree of the class's or -function's annotations, rather than their value. This enables users of -annotations to inspect the annotation expression and evaluate it using -domain-specific semantics that might differ from simply evaluating a Python -expression. This does not only benefit people using annotations for type -hints but also supports other uses of annotations. - -Note that we do not propose adding direct support for any of the specific -type annotation changes or additions that are discussed in this PEP and restrict -this proposal to laying the necessary groundwork that enables them. -Such changes are mentioned only as motivating examples to show why the ``AST`` -format is needed. +To more easily understand this proposal, we'll work through it all with an +example. Let's say a function is defined as ``def func(arg: 1 | 2): ...``. +Currently, its ``__annotate__`` method is generated to look (if it were Python) +like this:: + + def __annotate__(self, format): + if format >= 3: + raise NotImplementedError + return { + "arg": 1 | 2, + } + +We can see that it just checks whether the requested format is one of the two +``VALUE`` formats (for more detail on why two such formats exist, see +:pep:`749<749#adding-the-value-with-fake-globals-format>`) and returns a dict +mapping the argument name to its annotation. +Note here that while we see the string ``"1 | 2"`` in the source code, the +caller of the function never gets to see that. It merely receives the object +that expression evaluates to, the ``int`` object ``3``. + +The AST Format +============== + +In this PEP, we address this issue by proposing to add a new format, ``AST``. +It instructs ``__annotate__`` methods to return the abstract syntax tree of the +annotation expressions, rather than their value. This will let consumers of +annotations evaluate the AST using whatever semantics are appropriate to their +domain rather than the built-in Python expression semantics. + +Going back to our example, ``func`` will then have an ``__annotate__`` method +that conceptually might look something like this:: + + def __annotate__(self, format): + if format <= 2: + return { + "arg": 1 | 2, + } + elif format == 5: + return { + "arg": BinOp( + left=Constant(value=1), + op=BitOr(), + right=Constant(value=2), + ), + } + else: + raise NotImplementedError + +The newly supported format ``5`` will be the value of the ``AST`` format. When +it is passed, the function returns a similar dictionary that contains the +annotation's AST, as represented by the existing classes from the :py:mod:`ast` +module. Users will then be able to analyze the full annotation by inspecting +this AST object. + +!!!! TUPLES !!!! + +Analyzing the AST +================= + +The AST format intentionally leaves open many possibilities for users to treat +the returned AST objects in domain-specific ways. That being said, there are +some special considerations afforded to the most popular usage of annotations, +type hints. In particular, the :py:mod:`typing` module will contain helper +functions that internally use the ``AST`` format to retreive annotations and +then parse the syntax trees into familiar typing objects. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +In our example, we are using type annotations and are presuming that the typing +spec will have been changed to let bare integer literals refer to the literal +type containing them. + + + + + + + + + +We do not propose adding direct support for any of the specific type annotation +changes or additions that are discussed in this PEP and restrict this proposal +to laying the necessary groundwork that enables them. Such changes are mentioned +only as motivating examples to show why the ``AST`` format is needed. In order to support such future additions to the typing spec, the ``typing.get_type_hints`` helper function will support a mode that uses the From 6295d09c3f21e5c2b5488c27d381d2b0ea1da0a5 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:15 +0200 Subject: [PATCH 15/25] update to new implementation --- peps/pep-9999.rst | 591 +++++++++++++++++++++------------------------- 1 file changed, 263 insertions(+), 328 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 9c8e24e4d1f..299906a4edf 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -1,28 +1,27 @@ -PEP: 9999 -Title: AST Format for Annotation Functions +PEP: 849 +Title: More Expressive Type Expressions Author: Imogen Hergeth Sponsor: Jelle Zijlstra Status: Draft Type: Standards Track Topic: Typing Requires: 749 -Created: 17-Dec-2025 -Python-Version: 3.15 -Post-History: `05-Nov-2025 `__ +Created: 05-Sep-2026 +Python-Version: 3.16 +Post-History: `02-Sep-2026 `__ ******** Abstract ******** -This PEP proposes that a new value, ``Format.AST``, is added to the -:py:mod:`annotationlib` enum and associated protocol. It instructs annotation -functions to not evaluate the annotation expressions directly, but rather to -return the abstract syntax trees that define each of them. This lets runtime -consumers of annotations observe their full definition, rather than just the -object they evaluate to. Which, in turn, creates the possibility of simplifying -many existing type annotations and using existing intuitive Python syntax in -typing contexts. +Currently, only a small subset of Python expressions can be used in type +annotations, which greatly limits the expressiveness of the Python type system +and its ease of use. This PEP proposes to expand annotation functions such that +any Python expression can be used in type annotations. It does this by creating +a new annotation format which instructs annotation functions to return the +annotation's AST and namespaces. Runtime consumers of type annotations can then +evaluate these into typing objects. .. _motivation: @@ -30,237 +29,220 @@ typing contexts. Motivation ********** -Background -========== - -When annotations were introduced to Python in :pep:`3107` and :pep:`526`, they -were defined simply as expressions that relate to a variable and are stored in -a new special dictionary ``__annotations__``. That is, writing -``var_name: expression`` just causes the Python interpreter to evaluate -``expression`` and store the result in the enclosing function's or class's -``__annotations__`` dictionary under the ``var_name`` key. - -This approach not only makes the implementation very simple, it also gives users -the freedom to use annotations to attach various kinds of metadata to variables. -One possibility are type hints, which specify what sets of values a variable is -expected to contain and how a function can safely be called. These have proven -to be very popular and have first received official support in :pep:`484` and -many additional features since. - -But this implementation also creates challenges for type hints. While simple -types such as the set of all integers can be spelled by just writing -``var: int``, there is no built-in syntax for more complicated typing concepts -like generics. When :pep:`484<484#generics>` introduced these, it thus had to -repurpose an existing form of expressions, namely indexing, and define the -needed operators on classes that should be usable in generic type annotations. -Note that while conceptually the expressions ``some_dict["key"]`` and -``list[int]`` denote very different operations, dictionary indexing and -specialization of a generic type, Python treats them exactly the same, an -invocation of the ``__getitem__`` operator. - -:pep:`649` and :pep:`749` changed this behavior in some ways. Instead of the -``__annotations__`` dictionary being built as the annotations are encountered, -the annotation's execution is now delayed until ``__annotations__`` is actually -accessed. Internally, a new method ``__annotate__`` is synthesized that creates -the dictionary by evaluating the class's or function's annotations. However, -what remains unchanged is that annotations are still treated as ordinary -expressions. They are evaluated just like any other expression in a different -context, their execution just is delayed until they are needed. - -Problems with the Current Approach -================================== - -This behavior can cause problems where users want to use an expression in -annotations in a way that clashes with the expression's usual behavior. For -example, consider literal types, i.e. types that consist of some particular set -of literal values. Currently, these are written as e.g. ``Literal[1]``. Many -users of type annotations would prefer to drop the redundant ``Literal[]``, the -context of it occurring in a type annotation already makes it clear that the -``1`` represents the literal type containing only ``1``. This also is reflected -in other languages, such as TypeScript, implementing literal types that way. - -The problem arises when these literal types are combined with other typing -constructs. For example, the union type ``1 | 2 | 3``, representing values that -are either ``1``, ``2`` or ``3``, cannot be properly evaluated at runtime. We -would want it to evaluate to a ``UnionType`` containing references to -``1, 2, 3``. But since the ``__or__`` special method is already implemented on -``int`` as the bitwise or operation, it will just be evaluated to ``3``. There -is no way of recovering the ``1`` and ``2`` present in the annotation from the -``__annotation__`` dictionary. - -Similar issues prevent us from using display syntax to denote the built-in -container types, forcing us to write ``set[int]``, ``list[int]`` and -``dict[int, str]`` instead of just ``{int}``, ``[int]`` and ``{int: str}``. -For example, in the first case the display syntax causes a ``set`` object to be -constructed rather than a special object representing a type. And ``set`` -already implements ``__or__`` to create a new set containing all elements of -both arguments, rather than a ``Union`` object. - -While all of these examples do have existing workarounds, the -additional work required to spell these types and their readability issues do -present real problems. Many Python users, particularly those new to typing, -intuitively reach towards the easier and shorter syntax like ``(int, str)`` -to denote a tuple type. The more verbose syntax also is cumbersome to understand -when it occurs as a type argument for a generic type. A very common example are -matrix libraries like numpy that use integer tuples to define a matrix's shape. -When creating such a matrix, you simply write -``ndarray(..., shape=(16, 1000))``. But that matrix's type is spelled as -``ndarray[tuple[Literal[16], Literal[1000]], ...]``. A user unfamiliar with -the internals of the Python type system is hard-pressed to see such an -annotation and understand what information it is trying to tell them and why -they have to use these seemingly redundant ``Literal`` tags. - -This example also shows that the annotation specification's intention of them -being treated as any other expression is not being honored very well by this -implementation. While an annotation can contain arbitrary expressions, this is -only true from the interpreter's point of view. The vast majority of users, -which use type annotations have to switch from the familiar ``(16, 1000)`` -syntax to the annotation-specific ``tuple[Literal[16], Literal[1000]]``. - -There also are new typing features that are being held back by this requirement -to create syntactical workarounds. For example, inline typed dictionaries could -intuitively be defined inline as ``{"some_key": int, "other_key": str}``. But -this again does not work because the semantics of ``dict`` objects to not work -properly in typing contexts. There is `previous discussion -`__ -of this, with the main hurdle to implementation being the runtime behavior. -Other examples are `conditional types -`__, -which let users write ternary statements in type definitions, like -``type SomeAlias[T] = int if issubclass(T, str) else str``. Or extending integer -literal types to support basic arithmetic operations. This would enable array -libraries to properly track shapes across operations like concatenation. The -existing `work on this -`__ -unfortunately concluded that while this is a very useful feature for many users, -implementing it with current semantics is too verbose and cumbersome to gain -much traction. +Python type annotations are expressions attached to variable names that denote +which values can be assigned to a variable. They are not only available to +linters and type checkers via the source code directly, but also to runtime +reflection uses because the interpreter creates special annotation functions that +compute the objects the annotation expressions evaluate to. + +While this approach works well for many existing annotations and is easy to +understand, it also greatly limits the kinds of expressions that can be used +as type annotations. In the abstract, this is because the annotation functions +evaluate such expressions using the same mechanism that Python uses for regular +value expressions and in many cases those semantics are not compatible with +runtime reflection needs. + +We will now provide several more concrete examples of expressions that could +see use in type annotations but are currently not feasible. However, we are not +directly advocating for any specific such usage and this PEP does not implement +any of them. Rather, we are providing the groundwork that would make them and +similar future proposals possible. + +:pep:`586` introduced literal types, which enumerate a concrete list of possible +literal values, e.g. the numbers ``1``, ``2`` and ``3``. This is currently spelled +as ``Literal[1, 2, 3]``. Many users intuitively want to instead spell this as +``1 | 2 | 3``, writing out the bare literals and using the union operator to join +them, which also is what other languages such as TypeScript use. This is +currently not possible in Python because the expression ``1 | 2 | 3`` evaluates +simply to ``3`` because ``|`` is interpreted as the binary-or operation rather than +the union of types. Further, constant folding completely eliminates this +expression from appearing anywhere in the generated bytecode, and it thus is +completely impossible to retreive the actual type annotation at runtime. + +The above example only prevents us from using slightly shorter spelling for +already existing types. There also are many type annotations that either +are reasonably possible or even currently being discussed, but which are +infeasible to implement currently. The currently open draft of :pep:`827` +introduces many such types. For example, it proposes conditional types which +are type expressions that evaluate to one of two types depending on some +condition. Intuitively, these would be written as +``FirstType if TypeCondition else OtherType``, just like ternary expressions. +But that is not possible because no matter what ``TypeCondition`` evaluates to +at runtime, one of the other type expressions will never be evaluated and thus +be invisible to runtime introspection. + +Similar issues arise when defining other types that are defined using other +existing types. For example, one might want to write +``{K: NotRequired[T] for K, T in SomeTypedDict}`` to define a typed dict that has +the same definition as an existing typed dict, but where every key is optional. +This is currently not possible because type objects cannot be iterated over in +this way. This has also already been discussed in e.g. `this thread +`. + ********* Rationale ********* -To more easily understand this proposal, we'll work through it all with an -example. Let's say a function is defined as ``def func(arg: 1 | 2): ...``. -Currently, its ``__annotate__`` method is generated to look (if it were Python) -like this:: - - def __annotate__(self, format): - if format >= 3: - raise NotImplementedError - return { - "arg": 1 | 2, - } - -We can see that it just checks whether the requested format is one of the two -``VALUE`` formats (for more detail on why two such formats exist, see -:pep:`749<749#adding-the-value-with-fake-globals-format>`) and returns a dict -mapping the argument name to its annotation. -Note here that while we see the string ``"1 | 2"`` in the source code, the -caller of the function never gets to see that. It merely receives the object -that expression evaluates to, the ``int`` object ``3``. - -The AST Format +Overview +======== + +We propose to make it possible to use arbitrary expressions in type expressions +while maintaining full runtime introspection capabilities by introducting a new +format for annotation functions. It will instruct them to return objects that +contain both the annotations' ASTs and their namespace. These can then be used +to construct the actual type objects that runtime introspection users are +interested in. + +For example, consider the following class: +``` +class MyClass: + a: int + b: list[str] +``` + +When calling ``MyClass.__annotate__(Format.VALUE)`` it will still return the +usual annotation dict ``{"a": int, "b": list[str]}``. But when called as +``MyClass.__annotate__(Format.AST)`` we receive this dictionary: + +``` +{ + "a": AnnotationAST(ast.Name("int"), {"int": int}), + "b": AnnotationAST(ast.Subscript(ast.Name("list"), ast.Name("str")), {"list": list, "str": str), +} +``` + +However, most users will never see these objects directly. Rather, we propose +to add a new function ``get_type_annotations`` to the :py:mod:`typing` +module, which will internally perform the above call and then return the +familiar ``{"a": int, "b": list[str]}`` annotation dictionary. + +The power of this approach is that it lets us implement new type expressions +such as the ones mentioned above by simply extending the evaluation logic in +`get_type_annotations`. This logic can then be completely decoupled from the +usual Python expression semantics and can instead create typing objects that +allow complete runtime introspection. + +We also propose to add some additional utility functionality related to +annotation functions and these AST objects. In particular, a new +`create_annoate_function` in the :py:mod:`annotationlib` module to easily +synthesize an annotation function. The core of this functionality is already +implemented in the :py:mod:`dataclasses` module, and we foresee that many +users will need this in order to create annotation functions that support +this somewhat more complex format. + + +Implementation ============== -In this PEP, we address this issue by proposing to add a new format, ``AST``. -It instructs ``__annotate__`` methods to return the abstract syntax tree of the -annotation expressions, rather than their value. This will let consumers of -annotations evaluate the AST using whatever semantics are appropriate to their -domain rather than the built-in Python expression semantics. - -Going back to our example, ``func`` will then have an ``__annotate__`` method -that conceptually might look something like this:: - - def __annotate__(self, format): - if format <= 2: - return { - "arg": 1 | 2, - } - elif format == 5: - return { - "arg": BinOp( - left=Constant(value=1), - op=BitOr(), - right=Constant(value=2), - ), - } - else: - raise NotImplementedError - -The newly supported format ``5`` will be the value of the ``AST`` format. When -it is passed, the function returns a similar dictionary that contains the -annotation's AST, as represented by the existing classes from the :py:mod:`ast` -module. Users will then be able to analyze the full annotation by inspecting -this AST object. - -!!!! TUPLES !!!! - -Analyzing the AST -================= - -The AST format intentionally leaves open many possibilities for users to treat -the returned AST objects in domain-specific ways. That being said, there are -some special considerations afforded to the most popular usage of annotations, -type hints. In particular, the :py:mod:`typing` module will contain helper -functions that internally use the ``AST`` format to retreive annotations and -then parse the syntax trees into familiar typing objects. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -In our example, we are using type annotations and are presuming that the typing -spec will have been changed to let bare integer literals refer to the literal -type containing them. - - - - - - - - - -We do not propose adding direct support for any of the specific type annotation -changes or additions that are discussed in this PEP and restrict this proposal -to laying the necessary groundwork that enables them. Such changes are mentioned -only as motivating examples to show why the ``AST`` format is needed. +While the exact mechanism annotation functions use is an implementation detail, +it may be useful to look at what that might look like. Consider for example +the class from above: + +``` +MyClass: + a: int + b: list[str] +``` + +Under this proposal its annotation function may behave similar to this Python code: +``` +def __annotate__(format): + namespace = { + "int": int, + "list": list, + "str": str, + } + asts = { + "a": "\x1a\x03int", + "b": "\x18\x1a\x04list\x1a\x03str", + } + return _make_annotate_asts(namespace, asts) +``` + +It functions in essentially three steps. It populates the namespace dictionary +by performing ordinary variable name lookups. This is important to ensure that +we can later evaluate the returned AST objects using the correct bindings for +each variable. It also loads some string constants that contain binary data +defining each annotation's AST. This is a compact format that is easily +implemented with existing compiler functionality. It also is much faster to +parse than storing the source code of the annotations directly. The annotate +function then uses a new intrinsic to actually build the required annotation +AST objects. + +The proposed ``get_type_annotations`` function would then compute the type objects +using a function like this: + +``` +def eval_type_annotation_AST(ast, namespace): + match ast: + case ast.Name: + return namespace[ast.id] + case ast.Attribute: + value = eval_type_annotation_AST(ast.value, namespace) + return getattr(value, ast.attr) + ... +``` + + +Performance Impact +================== -In order to support such future additions to the typing spec, the -``typing.get_type_hints`` helper function will support a mode that uses the -``AST`` format to evaluate type annotations with the typing spec's semantics. +While every new feature has to be weighed against its impact on performance and +complexity, typing related features deserve additional scrutiy because type +annotations are an entirely optional part of the Python language. We thus need +to consider three seperate groups of users and its impact on them: users who +do not use type annotations at all, users who annotate their code for type +checkers and/or linters, but don't use runtime introspection and finally users +who also evaluate their type annotations at runtime. + +For this, we considered three metrics: the time it takes to import modules both +without and with type annotations, the annotation functions' size in memory and +finally the time it takes to actually evaluate the annotation functions. The +import time is most important for the first group of users, the second group is +also affected by the annotation functions' memory footprint and the time to +evaluate the annotation functions only impacts the last group of users. + +Using our reference implementation, we have found no significant difference in +import times of modules that do not use type annotations. For modules that do +use annotations, import was moderately faster and the memory footprint slightly +smaller using the proposed annotation functions, both by a few percent. But +unfortunately, evaluation time increases by a lot, about seven times as long. + +In total, since the negative performance impacts only affect the smallest group +of users, who also benefit from the newly possible type annotations, we consider +this a worthwhile trade off. The specification of the proposed format also is open +enough that many optimizations are possible should they be deemed necessary in +the future. For many users, this proposal will even be a slight performance +increase since their code never evaluates any annotation functions. + + +Usage in Other Places +===================== + +While this proposal enables the usage of future type expressions in annoations, +there also are other places where users want to write type expressions. For +example in ``cast(, value)``. Since the interpreter +cannot differentiate these cases from other function calls, it is impossible +for it to infer that it should use a mechanism like we suggest. + +We think that directly supporting these use cases is not needed. This would +require new syntax, which we do not think is merited for this functionality. +Instead, this problem can be avoided using an intermediate type alias. +That is, one would write: + +``` +type _TargetType = +cast(_TargetType, value) +``` + +This lets you use any new type expression within ```` +since type aliases also are implemented using annotation functions. While this +workaround is somewhat annoying, the fact that it is needed relatively +infrequently makes us believe that it is the better solution at this moment. +We recommend that this usage of intermediate type aliases is observed in the +future and new syntax to avoid it considered should it be necessary. -There are some places other than annotations where users need to spell a type, -for example, the first argument to ``typing.cast``. These use cases will be -covered by the existing practice of wrapping types in string literals. If such -a type needs to be introspected at runtime, a new helper function -``typing.eval_type`` can be used, which internally uses the new format to -evaluate a string under the typing spec's semantics. ************* Specification @@ -284,87 +266,41 @@ The ``AST`` Format A new value called ``AST`` is added to the ``annotationlib.Format`` enum with value 5. Annotation functions do not have to support this format. If an -annotation function is called with this format, it must return instances of -``ast.expr`` that do not contain ``ast.NamedExpr``, ``ast.Yield``, -``ast.YieldFrom`` or ``ast.Await`` nodes (the Python grammar -already prohibits the use of these features in annotations). - -Compiler-generated annotation functions will always support this format -and return the abstract syntax tree of the class's or function's annotations. -They will store the necessary AST data stored as tuple constants and then -construct a new ``ast.expr`` object each time they are called. These objects -will be functionally identical to the objects created by parsing the annotation -code directly. - -Annotate functions that are generated by library code will need to manually -support the new format or choose to raise ``NotImplementedError()`` when it -is requested. Most of these use cases synthesize new annotation functions by -introspecting existing function or class objects and combining and/or modifying -their return values. The new format makes this easier to do since the AST -objects can always be created regardless of the existence of forward references -or similar issues, they can be modified to denote the synthesized annotations -using existing tools, and they can be evaluated to create the objects requested -by other formats. +annotation function is called with this format, it must return a +``AnnotationAST`` object. These are instances of a proposed new class that +hold the annotation's AST as an ``ast.expr`` object and a namespace that is +used to evaluate them. + +Compiler-generated annotation functions will always support this format. +They will store the necessary AST data stored as string constants and then +construct a new ``AnnotationAST`` objects each time they are called. The +contained AST objects will be identical to the objects created by parsing the +annotation source code directly. + Helper Functions ================ -The existing ``typing.get_type_hints`` function will be modified to accept a -new, optional Boolean keyword argument ``use_type_syntax`` that defaults to -``False``. When set to true, ``get_type_hints`` does not call the underlying -annotation function with the requested format but always with ``Format.AST``. -The returned AST is then evaluated to an object, using the semantics of Python -expressions, modified by rules that are defined in the typing spec. At the time -this PEP is written, no such rules exist, and thus the AST is always evaluated -normally. - -When the keyword argument is set, any annotation expressions that are not -valid type annotation expressions, as `defined by the typing spec -__`, -cause ``get_type_hints`` to raise a ``SyntaxError()``. - -When the typing spec is changed to define new type expressions, -``get_type_hints`` will be modified accordingly and return a particular object -when it previously would have raised a ``SyntaxError`` on the same inputs. This -is an exemption to the usual deprecation policy and will be documented as such. -Other behavior of ``get_type_hints`` will not be affected; users can still rely -on the stability of returned values and other raised exceptions. - -A new helper method ``eval_type(tp: str, globals: Mapping[str, object], -locals: Mapping[str, object], format: Format = Format.VALUE) -> TypeForm`` -will be added to the :py:mod:`typing` module. It evaluates a string using the -same rules that ``get_type_hints`` uses with the type-specific semantics. -This function can be used to evaluate stringified types that are used in places -other than annotations. - -In order to ease the creation of synthesized annotation functions, a new helper -function ``create_annotate_from_asts(annotations: dict[str, tuple[ast.expr, -Mapping[str, object]]]) -> Callable[[Format], dict[str, object]]`` will be -added to -:py:mod:`annotationlib`. It creates an ``__annotate__`` method that supports -the ``VALUE``, ``FORWARDREF``, ``SOURCE`` and ``AST`` formats. The returned -values are the objects of the correct type that would be created if each of -the passed syntax trees were evaluated in the corresponding namespace. - -Pseudocode -========== -A function defined as ``def f(first: int, second: tuple[int, str])`` will have -a compiler-generated ``__annotate__`` method that might look something like this -were it written in Python:: - - def __annotate__(self, format): - if format <= 2: - return { - "first": int, - "second": tuple[int, str], - } - if format == 5: - from ast import _ast_from_tuple - return { - "first": _ast_from_tuple((26, "int", 1, 1, 1, 14)), - "second": _ast_from_tuple((...)), - } - raise NotImplementedError +A new function ``typing.get_type_annotations`` is added that functions similarly +to the existing ``annotationlib.get_annotations``, but instead calls the +underlying annotation function with ``Format.AST`` and then constructs typing +objects using the new ``typing.evaluate_type_ast`` helper. + +The existing ``typing.get_type_hints`` function will be deprecated. It has +slightly different semantics to both ``annotationlib.get_annotations`` and the +proposed function, which make it impossible to instead modify it to support the +new functionality. It also will be completely superfluous since users of type +annotations will need to call ``get_type_annotations`` instead to properly +resolve any type annotations that contain new typing features. Leaving this +function as-is will only create confusion about which function should be used. + +In order to simplify the creation of synthesized annotation functions, a new +helper function ``annotationlib.create_annotate_function`` will be added to +:py:mod:`annotationlib`. It accepts a mapping from variable names to annotation +objects in one of the existing annotation formats. Using these, it then returns +an annotation function that supports all annotation formats by converting the +passed in values appropriately. + *********************** Backwards Compatibility @@ -375,12 +311,12 @@ compatibility concerns. However, we also want to point out that future additions like the ones outlined in the :ref:`motivation` do not create backwards compatibility problems, even if adopted gradually. -Consider, for example, a change to the typing spec that made it so that -``var: 1`` is interpreted as a literal type ``Literal[1]``. When a user -evaluates the annotation method directly or with one of the helper methods, they +Consider, for example, a change to the typing spec that makes it so that +``var: 1`` is interpreted as a literal type ``Literal[1]``. Users that evaluate +the annotation method directly or with one of the existing helper methods, they will still observe the result as the plain integer ``1``. The changed semantics -only happen when the user opts in to the new semantics by calling -``typing.get_type_hints`` with the new keyword argument set. +are only considered when the user opts into them by calling the annotation +function with ``Format.AST`` or using ``typing.get_type_annotations``. In the most common use case, annotations are not consumed by the user directly but by some library code that introspects user-defined objects. Thus, adoption @@ -392,14 +328,6 @@ valid type annotations will not be changed to mean something different. Potential future changes can only define new semantics to syntax constructs that currently are not valid type annotations and thus do not occur in user code. -The only potential for issues to happen is if a user has written annotations -that currently are not legal type annotations but become legal with modified -semantics in the future. We believe that this case is exceedingly rare since -such users either do not intend to use annotations for type hints and will thus -not use a library that expects them to be type hints, or the code will already -not work since the annotations currently cannot be handled correctly by the -library code. - ********************* Security Implications @@ -411,6 +339,7 @@ object. Python also already offers the capability to access the source code and compiled byte code of introspected objects. The AST objects returned in the new format thus do not contain any previously unavailable information. + ***************** How to Teach This ***************** @@ -428,16 +357,22 @@ a place where it is statically known that a type form is expected. The new format and changes to helper functions will be documented as part of the language standard. Libraries that introspect type annotations will be able to -easily support any new type syntax by calling ``typing.get_type_hints`` and -should document this behavior so that their users are made aware of any +easily support any new type syntax by calling the provided utility functions +and should document this behavior so that their users are made aware of any potential future changes. +One potential issue is the :ref:`usage in other places` discussed in the +beginning. Since most users of type annotations will also use type checkers +and/or linters, we recommend that these tools implement checks for these errors +and suggest the fix via an intermediate type alias. + + ************************ Reference Implementation ************************ This proposal is prototyped in `a CPython fork -`__. +`__. ************** Rejected Ideas From ceb86729386c454415eeb5928335824d62e99943 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:52:15 +0200 Subject: [PATCH 16/25] rename pep --- peps/{pep-9999.rst => pep-849.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename peps/{pep-9999.rst => pep-849.rst} (100%) diff --git a/peps/pep-9999.rst b/peps/pep-849.rst similarity index 100% rename from peps/pep-9999.rst rename to peps/pep-849.rst From 75f87aee3d7d7cb93945d43a4fb43865cd018498 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 17:58:42 +0200 Subject: [PATCH 17/25] fix linting issues --- peps/{pep-849.rst => pep-0849.rst} | 97 +++++++++++++++--------------- 1 file changed, 50 insertions(+), 47 deletions(-) rename peps/{pep-849.rst => pep-0849.rst} (93%) diff --git a/peps/pep-849.rst b/peps/pep-0849.rst similarity index 93% rename from peps/pep-849.rst rename to peps/pep-0849.rst index 299906a4edf..eb3612f749d 100644 --- a/peps/pep-849.rst +++ b/peps/pep-0849.rst @@ -77,7 +77,7 @@ existing types. For example, one might want to write the same definition as an existing typed dict, but where every key is optional. This is currently not possible because type objects cannot be iterated over in this way. This has also already been discussed in e.g. `this thread -`. +`__. ********* @@ -95,22 +95,23 @@ to construct the actual type objects that runtime introspection users are interested in. For example, consider the following class: -``` -class MyClass: - a: int - b: list[str] -``` + +.. code-block:: python + + class MyClass: + a: int + b: list[str] When calling ``MyClass.__annotate__(Format.VALUE)`` it will still return the usual annotation dict ``{"a": int, "b": list[str]}``. But when called as ``MyClass.__annotate__(Format.AST)`` we receive this dictionary: -``` -{ - "a": AnnotationAST(ast.Name("int"), {"int": int}), - "b": AnnotationAST(ast.Subscript(ast.Name("list"), ast.Name("str")), {"list": list, "str": str), -} -``` +.. code-block:: python + + { + "a": AnnotationAST(ast.Name("int"), {"int": int}), + "b": AnnotationAST(ast.Subscript(ast.Name("list"), ast.Name("str")), {"list": list, "str": str), + } However, most users will never see these objects directly. Rather, we propose to add a new function ``get_type_annotations`` to the :py:mod:`typing` @@ -139,26 +140,27 @@ While the exact mechanism annotation functions use is an implementation detail, it may be useful to look at what that might look like. Consider for example the class from above: -``` -MyClass: - a: int - b: list[str] -``` +.. code-block:: python + + MyClass: + a: int + b: list[str] Under this proposal its annotation function may behave similar to this Python code: -``` -def __annotate__(format): - namespace = { - "int": int, - "list": list, - "str": str, - } - asts = { - "a": "\x1a\x03int", - "b": "\x18\x1a\x04list\x1a\x03str", - } - return _make_annotate_asts(namespace, asts) -``` + +.. code-block:: python + + def __annotate__(format): + namespace = { + "int": int, + "list": list, + "str": str, + } + asts = { + "a": "\x1a\x03int", + "b": "\x18\x1a\x04list\x1a\x03str", + } + return _make_annotate_asts(namespace, asts) It functions in essentially three steps. It populates the namespace dictionary by performing ordinary variable name lookups. This is important to ensure that @@ -173,16 +175,16 @@ AST objects. The proposed ``get_type_annotations`` function would then compute the type objects using a function like this: -``` -def eval_type_annotation_AST(ast, namespace): - match ast: - case ast.Name: - return namespace[ast.id] - case ast.Attribute: - value = eval_type_annotation_AST(ast.value, namespace) - return getattr(value, ast.attr) - ... -``` +.. code-block:: python + + def eval_type_annotation_AST(ast, namespace): + match ast: + case ast.Name: + return namespace[ast.id] + case ast.Attribute: + value = eval_type_annotation_AST(ast.value, namespace) + return getattr(value, ast.attr) + ... Performance Impact @@ -216,9 +218,10 @@ enough that many optimizations are possible should they be deemed necessary in the future. For many users, this proposal will even be a slight performance increase since their code never evaluates any annotation functions. +.. _outside-annos: -Usage in Other Places -===================== +Usage Outside of Annotations +============================ While this proposal enables the usage of future type expressions in annoations, there also are other places where users want to write type expressions. For @@ -231,10 +234,10 @@ require new syntax, which we do not think is merited for this functionality. Instead, this problem can be avoided using an intermediate type alias. That is, one would write: -``` -type _TargetType = -cast(_TargetType, value) -``` +.. code-block:: python + + type _TargetType = + cast(_TargetType, value) This lets you use any new type expression within ```` since type aliases also are implemented using annotation functions. While this @@ -361,7 +364,7 @@ easily support any new type syntax by calling the provided utility functions and should document this behavior so that their users are made aware of any potential future changes. -One potential issue is the :ref:`usage in other places` discussed in the +One potential issue is the :ref:`outside-annos` discussed in the beginning. Since most users of type annotations will also use type checkers and/or linters, we recommend that these tools implement checks for these errors and suggest the fix via an intermediate type alias. From e0fb0253df20dbe4ede90f3523129ca32e3a3bbe Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 18:12:45 +0200 Subject: [PATCH 18/25] fix spelling --- peps/pep-0849.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index eb3612f749d..5b2a855e9c3 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -57,7 +57,7 @@ currently not possible in Python because the expression ``1 | 2 | 3`` evaluates simply to ``3`` because ``|`` is interpreted as the binary-or operation rather than the union of types. Further, constant folding completely eliminates this expression from appearing anywhere in the generated bytecode, and it thus is -completely impossible to retreive the actual type annotation at runtime. +completely impossible to retrieve the actual type annotation at runtime. The above example only prevents us from using slightly shorter spelling for already existing types. There also are many type annotations that either @@ -76,7 +76,7 @@ existing types. For example, one might want to write ``{K: NotRequired[T] for K, T in SomeTypedDict}`` to define a typed dict that has the same definition as an existing typed dict, but where every key is optional. This is currently not possible because type objects cannot be iterated over in -this way. This has also already been discussed in e.g. `this thread +this way. This has also already been discussed in e.g. `this thread `__. @@ -88,7 +88,7 @@ Overview ======== We propose to make it possible to use arbitrary expressions in type expressions -while maintaining full runtime introspection capabilities by introducting a new +while maintaining full runtime introspection capabilities by introducing a new format for annotation functions. It will instruct them to return objects that contain both the annotations' ASTs and their namespace. These can then be used to construct the actual type objects that runtime introspection users are @@ -120,13 +120,13 @@ familiar ``{"a": int, "b": list[str]}`` annotation dictionary. The power of this approach is that it lets us implement new type expressions such as the ones mentioned above by simply extending the evaluation logic in -`get_type_annotations`. This logic can then be completely decoupled from the +``get_type_annotations``. This logic can then be completely decoupled from the usual Python expression semantics and can instead create typing objects that allow complete runtime introspection. We also propose to add some additional utility functionality related to annotation functions and these AST objects. In particular, a new -`create_annoate_function` in the :py:mod:`annotationlib` module to easily +``create_annoate_function`` in the :py:mod:`annotationlib` module to easily synthesize an annotation function. The core of this functionality is already implemented in the :py:mod:`dataclasses` module, and we foresee that many users will need this in order to create annotation functions that support @@ -193,7 +193,7 @@ Performance Impact While every new feature has to be weighed against its impact on performance and complexity, typing related features deserve additional scrutiy because type annotations are an entirely optional part of the Python language. We thus need -to consider three seperate groups of users and its impact on them: users who +to consider three separate groups of users and its impact on them: users who do not use type annotations at all, users who annotate their code for type checkers and/or linters, but don't use runtime introspection and finally users who also evaluate their type annotations at runtime. @@ -223,7 +223,7 @@ increase since their code never evaluates any annotation functions. Usage Outside of Annotations ============================ -While this proposal enables the usage of future type expressions in annoations, +While this proposal enables the usage of future type expressions in annotations, there also are other places where users want to write type expressions. For example in ``cast(, value)``. Since the interpreter cannot differentiate these cases from other function calls, it is impossible From d495016d4b4cd383ebebbb946356a3963499ca4d Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 18:40:47 +0200 Subject: [PATCH 19/25] update rejected ideas --- peps/pep-0849.rst | 43 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 9 deletions(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index 5b2a855e9c3..a7a0eec2a90 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -381,16 +381,41 @@ This proposal is prototyped in `a CPython fork Rejected Ideas ************** -A New Argument for Annotation Functions -======================================= - -An initial idea was to add a new optional argument to annotation functions, -which instructs them to return objects of the specified format's type but -using type annotation-specific semantics. This was rejected since it means that -the type-specific semantics need to be defined in the interpreter itself, which -limits the kinds of semantics that are possible. It also locks the usage of +Creating Typing Objects Within Annotation Functions +=================================================== + +An initial idea was to modify annotation functions such that they create the +relevant typing objects themselves. This can be achieved in several ways, for +example via a new optional argument to signal typing-specific semantics or a new +format. These approaches were rejected because they force the type-specific +semantics to be defined in the interpreter itself. This not only limits the +semantics to e.g. not require namespace lookups and also locks the usage of typing features to the Python version that is being used, rather than allowing -the usual backporting via ``typing_extensions``. +the currently possible backporting via ``typing_extensions``. + +Storing Annotation Source Code +============================== + +Instead of storing binary data that defines the annotations' ASTs an alternative +is to simply store the annotations source code directly. This also was presented +as a possibility all the way back in :pep:`649`. The two approaches are largely +equivalent since one can create the AST from the source code and vice versa. + +In a performance comparison the two data representations also are largely +equivalent concerning import times and memory usage. However, parsing the source +code to then create typing objects is significantly slower than working with the +AST data, by a factor of 3. Storing the source code also greatly limits future +optimizations since the data representation is directly exposed as the API. + +Not Returning AST Namespaces +============================ + +In order to properly evaluate an AST into the correct typing objects, the +evaluating function needs to have access to the namespace the annotation was +defined in. Initially it seems like this namespace can be reconstructed from +the annotate function's object since they contain the globals and cellvars used. +However, this is not sufficient since namespaces can use more complex lookup +logic when ``global`` statements or name mangling are used. ********* Copyright From 4639050a1db75ef0366332708e2bae849bf81b7f Mon Sep 17 00:00:00 2001 From: Imogen Date: Sat, 19 Sep 2026 18:41:45 +0200 Subject: [PATCH 20/25] annotation functions -> annotate functions --- peps/pep-0849.rst | 52 +++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index a7a0eec2a90..513e1029bbb 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -17,9 +17,9 @@ Abstract Currently, only a small subset of Python expressions can be used in type annotations, which greatly limits the expressiveness of the Python type system -and its ease of use. This PEP proposes to expand annotation functions such that +and its ease of use. This PEP proposes to expand annotate functions such that any Python expression can be used in type annotations. It does this by creating -a new annotation format which instructs annotation functions to return the +a new annotation format which instructs annotate functions to return the annotation's AST and namespaces. Runtime consumers of type annotations can then evaluate these into typing objects. @@ -32,12 +32,12 @@ Motivation Python type annotations are expressions attached to variable names that denote which values can be assigned to a variable. They are not only available to linters and type checkers via the source code directly, but also to runtime -reflection uses because the interpreter creates special annotation functions that +reflection uses because the interpreter creates special annotate functions that compute the objects the annotation expressions evaluate to. While this approach works well for many existing annotations and is easy to understand, it also greatly limits the kinds of expressions that can be used -as type annotations. In the abstract, this is because the annotation functions +as type annotations. In the abstract, this is because the annotate functions evaluate such expressions using the same mechanism that Python uses for regular value expressions and in many cases those semantics are not compatible with runtime reflection needs. @@ -89,7 +89,7 @@ Overview We propose to make it possible to use arbitrary expressions in type expressions while maintaining full runtime introspection capabilities by introducing a new -format for annotation functions. It will instruct them to return objects that +format for annotate functions. It will instruct them to return objects that contain both the annotations' ASTs and their namespace. These can then be used to construct the actual type objects that runtime introspection users are interested in. @@ -125,18 +125,18 @@ usual Python expression semantics and can instead create typing objects that allow complete runtime introspection. We also propose to add some additional utility functionality related to -annotation functions and these AST objects. In particular, a new +annotate functions and these AST objects. In particular, a new ``create_annoate_function`` in the :py:mod:`annotationlib` module to easily -synthesize an annotation function. The core of this functionality is already +synthesize an annotate function. The core of this functionality is already implemented in the :py:mod:`dataclasses` module, and we foresee that many -users will need this in order to create annotation functions that support +users will need this in order to create annotate functions that support this somewhat more complex format. Implementation ============== -While the exact mechanism annotation functions use is an implementation detail, +While the exact mechanism annotate functions use is an implementation detail, it may be useful to look at what that might look like. Consider for example the class from above: @@ -146,7 +146,7 @@ the class from above: a: int b: list[str] -Under this proposal its annotation function may behave similar to this Python code: +Under this proposal its annotate function may behave similar to this Python code: .. code-block:: python @@ -199,16 +199,16 @@ checkers and/or linters, but don't use runtime introspection and finally users who also evaluate their type annotations at runtime. For this, we considered three metrics: the time it takes to import modules both -without and with type annotations, the annotation functions' size in memory and -finally the time it takes to actually evaluate the annotation functions. The +without and with type annotations, the annotate functions' size in memory and +finally the time it takes to actually evaluate the annotate functions. The import time is most important for the first group of users, the second group is -also affected by the annotation functions' memory footprint and the time to -evaluate the annotation functions only impacts the last group of users. +also affected by the annotate functions' memory footprint and the time to +evaluate the annotate functions only impacts the last group of users. Using our reference implementation, we have found no significant difference in import times of modules that do not use type annotations. For modules that do use annotations, import was moderately faster and the memory footprint slightly -smaller using the proposed annotation functions, both by a few percent. But +smaller using the proposed annotate functions, both by a few percent. But unfortunately, evaluation time increases by a lot, about seven times as long. In total, since the negative performance impacts only affect the smallest group @@ -216,7 +216,7 @@ of users, who also benefit from the newly possible type annotations, we consider this a worthwhile trade off. The specification of the proposed format also is open enough that many optimizations are possible should they be deemed necessary in the future. For many users, this proposal will even be a slight performance -increase since their code never evaluates any annotation functions. +increase since their code never evaluates any annotate functions. .. _outside-annos: @@ -240,7 +240,7 @@ That is, one would write: cast(_TargetType, value) This lets you use any new type expression within ```` -since type aliases also are implemented using annotation functions. While this +since type aliases also are implemented using annotate functions. While this workaround is somewhat annoying, the fact that it is needed relatively infrequently makes us believe that it is the better solution at this moment. We recommend that this usage of intermediate type aliases is observed in the @@ -251,7 +251,7 @@ future and new syntax to avoid it considered should it be necessary. Specification ************* -Throughout this PEP, when we talk about *annotation functions/methods*, we are +Throughout this PEP, when we talk about *annotate functions/methods*, we are referring to both the ``__annotate__`` special methods found on some objects and the following methods found on typing objects: @@ -268,13 +268,13 @@ The ``AST`` Format ================== A new value called ``AST`` is added to the ``annotationlib.Format`` enum with -value 5. Annotation functions do not have to support this format. If an -annotation function is called with this format, it must return a +value 5. Annotate functions do not have to support this format. If an +annotate function is called with this format, it must return a ``AnnotationAST`` object. These are instances of a proposed new class that hold the annotation's AST as an ``ast.expr`` object and a namespace that is used to evaluate them. -Compiler-generated annotation functions will always support this format. +Compiler-generated annotate functions will always support this format. They will store the necessary AST data stored as string constants and then construct a new ``AnnotationAST`` objects each time they are called. The contained AST objects will be identical to the objects created by parsing the @@ -286,7 +286,7 @@ Helper Functions A new function ``typing.get_type_annotations`` is added that functions similarly to the existing ``annotationlib.get_annotations``, but instead calls the -underlying annotation function with ``Format.AST`` and then constructs typing +underlying annotate function with ``Format.AST`` and then constructs typing objects using the new ``typing.evaluate_type_ast`` helper. The existing ``typing.get_type_hints`` function will be deprecated. It has @@ -297,11 +297,11 @@ annotations will need to call ``get_type_annotations`` instead to properly resolve any type annotations that contain new typing features. Leaving this function as-is will only create confusion about which function should be used. -In order to simplify the creation of synthesized annotation functions, a new +In order to simplify the creation of synthesized annotate functions, a new helper function ``annotationlib.create_annotate_function`` will be added to :py:mod:`annotationlib`. It accepts a mapping from variable names to annotation objects in one of the existing annotation formats. Using these, it then returns -an annotation function that supports all annotation formats by converting the +an annotate function that supports all annotation formats by converting the passed in values appropriately. @@ -381,10 +381,10 @@ This proposal is prototyped in `a CPython fork Rejected Ideas ************** -Creating Typing Objects Within Annotation Functions +Creating Typing Objects Within Annotate Functions =================================================== -An initial idea was to modify annotation functions such that they create the +An initial idea was to modify annotate functions such that they create the relevant typing objects themselves. This can be achieved in several ways, for example via a new optional argument to signal typing-specific semantics or a new format. These approaches were rejected because they force the type-specific From c849f87cc38be37f1dc18deb0df8d06e6345c093 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sun, 20 Sep 2026 10:23:40 +0200 Subject: [PATCH 21/25] fix PEP header --- .github/CODEOWNERS | 2 ++ peps/pep-0849.rst | 9 +++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 13e74a486cb..2007070652d 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -722,6 +722,8 @@ peps/pep-0844.rst @warsaw peps/pep-0845.rst @ethanfurman peps/pep-0846.rst @JelleZijlstra @johnslavik peps/pep-0847.rst @dstufft +# peps/pep-0848.rst +peps/pep-0849.rst @JelleZijlstra # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index 513e1029bbb..7e21c1b4a7d 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -1,14 +1,15 @@ PEP: 849 Title: More Expressive Type Expressions -Author: Imogen Hergeth +Author: Imogen Hergeth Sponsor: Jelle Zijlstra +Discussions-To: Pending Status: Draft Type: Standards Track Topic: Typing -Requires: 749 -Created: 05-Sep-2026 +Created: 20-Sep-2026 Python-Version: 3.16 -Post-History: `02-Sep-2026 `__ +Post-History: `05-Nov-2025 `__ + `02-Sep-2026 `__ ******** From 4abe9e109bfa0d5e5cd5fe93c1e577171fef8858 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sun, 20 Sep 2026 16:39:22 +0200 Subject: [PATCH 22/25] incorporate feedback --- peps/pep-0849.rst | 49 +++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index 7e21c1b4a7d..fd743107b91 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -16,9 +16,15 @@ Post-History: `05-Nov-2025 , value)``. Since the interpreter cannot differentiate these cases from other function calls, it is impossible for it to infer that it should use a mechanism like we suggest. -We think that directly supporting these use cases is not needed. This would -require new syntax, which we do not think is merited for this functionality. -Instead, this problem can be avoided using an intermediate type alias. -That is, one would write: +This problem can be avoided using an intermediate type alias: .. code-block:: python @@ -241,11 +256,13 @@ That is, one would write: cast(_TargetType, value) This lets you use any new type expression within ```` -since type aliases also are implemented using annotate functions. While this -workaround is somewhat annoying, the fact that it is needed relatively -infrequently makes us believe that it is the better solution at this moment. -We recommend that this usage of intermediate type aliases is observed in the -future and new syntax to avoid it considered should it be necessary. +since type aliases also are implemented using annotate functions. + +While this solution only presents a workaround to this problem, a more +comprehensive fix would require adding a new keyword to the Python language, +which we do not think is necessary. This decision can be reconsidered in the +future if using intermediate type aliases like this does present a significant +problem in real-world code. ************* From 63b22b0daf322c076641c1837c9d570e2c5b29cc Mon Sep 17 00:00:00 2001 From: Imogen Date: Sun, 20 Sep 2026 18:03:54 +0200 Subject: [PATCH 23/25] add explanations around STRING format and backcompat --- peps/pep-0849.rst | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index fd743107b91..4598f20abd2 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -315,6 +315,14 @@ annotations will need to call ``get_type_annotations`` instead to properly resolve any type annotations that contain new typing features. Leaving this function as-is will only create confusion about which function should be used. +The AST format also provides a simpler and more reliable method to create +annotations in the ``STRING`` and ``FORWARDREF`` formats. Currently, compiler +generated annotate functions do not support these directly, rather the helper +functions in :py:mod:`annotationlib` try to create a best-effort AST and then +uparse it into the requested format. Under this PEP these helper functions will +instead use the AST format and unparse the result. This creates more accurate +results for these formats in many cases. + In order to simplify the creation of synthesized annotate functions, a new helper function ``annotationlib.create_annotate_function`` will be added to :py:mod:`annotationlib`. It accepts a mapping from variable names to annotation @@ -327,8 +335,16 @@ passed in values appropriately. Backwards Compatibility *********************** -Since this PEP only adds new functionality, there are no direct backwards -compatibility concerns. However, we also want to point out that future additions +The new annotation format and helper functions only add new functionality and +thus do not have backwards compatibility concerns. The deprecation of +``typing.get_type_hints`` means that existing code that uses this function will +break once it is removed from the standard library. We recommend that users +migrate to ``annotationlib.get_annotations`` or ``typing.get_type_annotations``, +depending on which semantics they want to use. While these functions have +slightly different behaviour in some cases, they are a drop-in replacement +most of the time. + +We further want to point out that future additions like the ones outlined in the :ref:`motivation` do not create backwards compatibility problems, even if adopted gradually. @@ -418,6 +434,9 @@ Instead of storing binary data that defines the annotations' ASTs an alternative is to simply store the annotations source code directly. This also was presented as a possibility all the way back in :pep:`649`. The two approaches are largely equivalent since one can create the AST from the source code and vice versa. +While the unparsed AST is not necessarily the exact string that occurred in +the source code since it can contain different whitespace, it is semantically +equivalent and in particular is not affected by compiler optimizations. In a performance comparison the two data representations also are largely equivalent concerning import times and memory usage. However, parsing the source From 8052375b5fad9e5d890a342c0a32075c52897332 Mon Sep 17 00:00:00 2001 From: Imogen Date: Sun, 20 Sep 2026 22:46:24 +0200 Subject: [PATCH 24/25] fix spelling/grammar mistakes --- peps/pep-0849.rst | 62 +++++++++++++++++++++++------------------------ 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index 4598f20abd2..b6ff66a1aee 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -17,7 +17,7 @@ Abstract ******** Currently, the Python interpreter allows any expression to be used as an -annotation. But their most popular usage, type anntotations, are restricted to +annotation. But their most popular usage, type annotations, are restricted to only a small subset of possible expressions. This is because type annotations need to be introspectable at runtime and many valid expressions produce values where that is not possible. This greatly limits which expressions can @@ -38,7 +38,7 @@ Motivation Python type annotations are expressions attached to variable names that denote which values can be assigned to a variable. They are not only available to -linters and type checkers via the source code directly, but also to runtime +linters and type checkers via the source code directly but also to runtime reflection uses because the interpreter creates special annotate functions that compute the objects the annotation expressions evaluate to. @@ -46,7 +46,7 @@ While this approach works well for many existing annotations and is easy to understand, it also greatly limits the kinds of expressions that can be used as type annotations. In the abstract, this is because the annotate functions evaluate such expressions using the same mechanism that Python uses for regular -value expressions and in many cases those semantics are not compatible with +value expressions, and in many cases those semantics are not compatible with runtime reflection needs. We will now provide several more concrete examples of expressions that could @@ -66,11 +66,11 @@ the union of types. Further, constant folding eliminates this expression from appearing anywhere in the generated bytecode, and it thus is impossible to retrieve the actual type annotation at runtime. -The above example only prevents us from using slightly shorter spelling for +The above example only prevents us from using slightly shorter spellings for already existing types. There also are many type annotations that either are reasonably possible or even currently being discussed, but which are infeasible to implement currently. The currently open draft of :pep:`827` -introduces many such types. For example, it proposes conditional types which +introduces many such types. For example, it proposes conditional types, which are type expressions that evaluate to one of two types depending on some condition. Intuitively, these would be written as ``FirstType if TypeCondition else OtherType``, just like ternary expressions. @@ -144,7 +144,7 @@ Implementation ============== While the exact mechanism annotate functions use is an implementation detail, -it may be useful to look at what that might look like. Consider for example +it may be useful to look at what that might look like. Consider, for example, the class from above: .. code-block:: python @@ -202,13 +202,13 @@ complexity, typing related features deserve additional scrutiny because type annotations are an entirely optional part of the Python language. We thus need to consider three separate groups of users and its impact on them: users who do not use type annotations at all, users who annotate their code for type -checkers and/or linters, but don't use runtime introspection and finally users +checkers and/or linters but don't use runtime introspection and finally, users who also evaluate their type annotations at runtime. For this, we considered three metrics: the time it takes to import modules both without and with type annotations, the annotate functions' size in memory and -finally the time it takes to actually evaluate the annotate functions. The -import time is most important for the first group of users, the second group is +the time it takes to actually evaluate the annotate functions. The +import time is most important for the first group of users; the second group is also affected by the annotate functions' memory footprint and the time to evaluate the annotate functions only impacts the last group of users. @@ -217,24 +217,24 @@ import times of modules that do not use type annotations. For modules that do use annotations, import was moderately faster and the memory footprint slightly smaller using the proposed annotate functions, both by a few percent. But unfortunately, evaluation time can increase significantly. In the worst case, -when annotations are requested in the value format and every used named is +when annotations are requested in the value format and every used name is defined, the increase is about sevenfold. However, when some names are not -defined and the string or forwardref formats have to be used the current -approach also is significantly slower and results in comparable times to the -proposed annotate functions. +defined and the ``STRING`` or ``FORWARDREF`` formats have to be used, the +current approach is also significantly slower and results in comparable times +to the proposed annotate functions. A common situation where type annotations are evaluated is when tools like :py:mod:`dataclasses` or similar ORM packages analyze class or function -definitions to synthesize additional behaviour. For these tools, the time to -e.g. create a dataclass will be impacted by this proposal. But as mentioned +definitions to synthesize additional behavior. For these tools, the time to, +e.g., create a dataclass will be impacted by this proposal. But as mentioned above, there already are many situations where inspecting annotations takes a similar amount of time. In total, since the negative performance impacts only affect the smallest group of users, who also benefit from the newly possible type annotations, we consider -this a worthwhile trade off. The specification of the proposed format also is open -enough that many optimizations are possible should they be deemed necessary in -the future. For many users, this proposal will even be a slight performance +this a worthwhile trade-off. The specification of the proposed format is also +open enough that many optimizations are possible should they be deemed necessary +in the future. For many users, this proposal will even be a slight performance increase since their code never evaluates any annotate functions. .. _outside-annos: @@ -316,19 +316,19 @@ resolve any type annotations that contain new typing features. Leaving this function as-is will only create confusion about which function should be used. The AST format also provides a simpler and more reliable method to create -annotations in the ``STRING`` and ``FORWARDREF`` formats. Currently, compiler -generated annotate functions do not support these directly, rather the helper -functions in :py:mod:`annotationlib` try to create a best-effort AST and then -uparse it into the requested format. Under this PEP these helper functions will -instead use the AST format and unparse the result. This creates more accurate -results for these formats in many cases. +annotations in the ``STRING`` and ``FORWARDREF`` formats. Currently, +compiler-generated annotate functions do not support these directly; rather, the +helper functions in :py:mod:`annotationlib` try to create a best-effort AST and +then unparse it into the requested format. Under this PEP these helper functions +will instead use the AST format and unparse the result. This creates more +accurate results for these formats in many cases. In order to simplify the creation of synthesized annotate functions, a new helper function ``annotationlib.create_annotate_function`` will be added to :py:mod:`annotationlib`. It accepts a mapping from variable names to annotation objects in one of the existing annotation formats. Using these, it then returns an annotate function that supports all annotation formats by converting the -passed in values appropriately. +passed-in values appropriately. *********************** @@ -341,7 +341,7 @@ thus do not have backwards compatibility concerns. The deprecation of break once it is removed from the standard library. We recommend that users migrate to ``annotationlib.get_annotations`` or ``typing.get_type_annotations``, depending on which semantics they want to use. While these functions have -slightly different behaviour in some cases, they are a drop-in replacement +slightly different behavior in some cases, they are a drop-in replacement most of the time. We further want to point out that future additions @@ -350,7 +350,7 @@ compatibility problems, even if adopted gradually. Consider, for example, a change to the typing spec that makes it so that ``var: 1`` is interpreted as a literal type ``Literal[1]``. Users that evaluate -the annotation method directly or with one of the existing helper methods, they +the annotation method directly or with one of the existing helper methods will still observe the result as the plain integer ``1``. The changed semantics are only considered when the user opts into them by calling the annotation function with ``Format.AST`` or using ``typing.get_type_annotations``. @@ -373,7 +373,7 @@ Security Implications There are no known security implications for this change. Calling annotation functions already could execute arbitrary code defined in the annotated object. Python also already offers the capability to access the source code and -compiled byte code of introspected objects. The AST objects returned in the +compiled bytecode of introspected objects. The AST objects returned in the new format thus do not contain any previously unavailable information. @@ -384,7 +384,7 @@ How to Teach This Users of annotations are not directly affected by this proposal. It intends to enable the simplification of the way type annotations are spelled; any future changes based on this PEP will need to be evaluated on their own merits. -Documentation will inform users that any new semantics is only natively +Documentation will inform users that any new semantics are only natively supported in annotations. Since this is by far the most common place for types to be spelled, we expect this to not be a big limitation. Other places where types can occur already require users to wrap forward references in string @@ -436,9 +436,9 @@ as a possibility all the way back in :pep:`649`. The two approaches are largely equivalent since one can create the AST from the source code and vice versa. While the unparsed AST is not necessarily the exact string that occurred in the source code since it can contain different whitespace, it is semantically -equivalent and in particular is not affected by compiler optimizations. +equivalent and, in particular, is not affected by compiler optimizations. -In a performance comparison the two data representations also are largely +In a performance comparison, the two data representations also are largely equivalent concerning import times and memory usage. However, parsing the source code to then create typing objects is significantly slower than working with the AST data, by a factor of 3. Storing the source code also greatly limits future From b5d9810168fbc624c642cd8b03beb57e0aedf9aa Mon Sep 17 00:00:00 2001 From: Imogen <59090860+ImogenBits@users.noreply.github.com> Date: Mon, 21 Sep 2026 09:17:45 +0200 Subject: [PATCH 25/25] Update peps/pep-0849.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0849.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst index b6ff66a1aee..e6bd3434403 100644 --- a/peps/pep-0849.rst +++ b/peps/pep-0849.rst @@ -435,7 +435,7 @@ is to simply store the annotations source code directly. This also was presented as a possibility all the way back in :pep:`649`. The two approaches are largely equivalent since one can create the AST from the source code and vice versa. While the unparsed AST is not necessarily the exact string that occurred in -the source code since it can contain different whitespace, it is semantically +the source code since the AST does not preserve the precise formatting of the code, it is semantically equivalent and, in particular, is not affected by compiler optimizations. In a performance comparison, the two data representations also are largely