Skip to content

Support free-threaded CPython builds - #7

Merged
tarebyte merged 3 commits into
masterfrom
fix/python-free-threading
Aug 31, 2026
Merged

Support free-threaded CPython builds#7
tarebyte merged 3 commits into
masterfrom
fix/python-free-threading

Conversation

@tarebyte

@tarebyte tarebyte commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Problem

The Python extension used single-phase initialization (PyModule_Create) and declared nothing about the GIL. On a free-threaded interpreter, importing a module that does not declare GIL support re-enables the GIL at runtime and emits a warning, which defeats the point of running that build.

Change

Follows the C API Extension Support for Free Threading how-to.

  • Multi-phase initialization. PyInit__binding returns PyModuleDef_Init(&module) instead of PyModule_Create(&module), with m_size changed from -1 to 0. The module holds no state, so there is nothing to migrate.
  • Py_mod_gil slot. Py_MOD_GIL_NOT_USED is declared through m_slots, which is the how-to's recommended form for multi-phase modules.
  • Stable ABI only on default builds. Py_mod_gil is not part of the limited API, so Py_LIMITED_API and py_limited_api are set only when the interpreter is not free-threaded, and the wheel tag stays cp39/abi3 in that case. A free-threaded build produces a version-specific wheel instead.
  • Py_GIL_DISABLED on Windows. The how-to notes that CPython defines this macro automatically on POSIX free-threaded builds but not on Windows, where it has to be passed to the compiler. Without it the #ifdef Py_GIL_DISABLED guard in binding.c is false, the slot is dropped, and a free-threaded Windows build silently produces a module that re-enables the GIL. setup.py now passes it explicitly whenever sysconfig.get_config_var("Py_GIL_DISABLED") is set. On POSIX that repeats the existing definition with the same value, which is a legal redefinition.

Validation

Verified on CPython 3.14.7 with the GIL enabled, which is the configuration this change must not disturb:

  • pip install . succeeds and still produces _binding.abi3.so, so the stable-ABI build is unchanged.
  • tree_sitter_ruby.language() returns a PyCapsule and the queries/*.scm files are still packaged.

Not verified: no free-threaded interpreter was available, so the free-threaded and Windows paths are derived from the CPython how-to rather than executed. What is directly checked here is that the conditionals leave the default build byte-for-byte equivalent in shape.

Reference

Reimplements tree-sitter#286, plus the Windows Py_GIL_DISABLED definition, which that PR does not include.

tarebyte and others added 3 commits August 24, 2026 00:16
The extension was always built against the stable ABI
(`Py_LIMITED_API` / `py_limited_api=True`). The limited API is not
available on free-threaded CPython 3.13+ builds, because the limited API
headers omit the atomic operations that thread-safe reference counting
needs, so the module failed to build there.

Only request the stable ABI (and the `abi3` wheel tag) when
`Py_GIL_DISABLED` is unset. Switch the module to multi-phase
initialisation so it can declare `Py_MOD_GIL_NOT_USED`; the binding only
returns a pointer to the static language table and holds no mutable
module state, so it does not need the GIL.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CPython defines this macro in pyconfig.h on POSIX free-threaded builds
but not on Windows, where it must be passed to the compiler. Without it
the guard around the Py_mod_gil slot in binding.c is false on Windows and
the module ends up re-enabling the GIL at import.

Hoist the free-threading check and the macro list to module level so the
three places that depend on it cannot drift apart.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ac21705f-befb-4606-9c83-657e3e9ea0dc

@espressolee espressolee left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed exact head ac79bb41506044d1f5fe346769d001f4c8a21b06 and do not see a blocking issue in this PR.

The two-file change is internally consistent: normal CPython builds retain the cp39-abi3 wheel path, while a free-threaded build drops the limited API, retains the interpreter-specific cp314-cp314t tag, and exposes Py_mod_gil = Py_MOD_GIL_NOT_USED through multi-phase initialization. I also checked the generated parser/scanner surface: parser tables are const and external scanner state is allocated per parser, so I did not find shared mutable state contradicting the marker in the exercised binding.

I independently built this exact head on Windows 11 x64 with CPython 3.14.7t. The result was a cp314-cp314t-win_amd64 wheel with a .cp314t-win_amd64.pyd; import kept the GIL disabled, and a 12-thread stress run completed 2400/2400 parses with zero errors. The exact-head wheel SHA256 was 14821a1d09227378cae991cf4e78e967844ca36d4dfa6c8955241eb0b0f63828. The base control failed at the expected limited-API/free-threaded incompatibility.

The only gap I see is nonblocking: the repository itself does not yet carry a free-threaded CI regression that asserts the GIL state and wheel tag. My approval is scoped to this exact head and the tested Windows x64/CPython 3.14.7t surface.

@tarebyte
tarebyte merged commit 1a4bd0b into master Aug 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants