Ship only the TensorRT delegate in the ExecuTorch runtime wheel by shoumikhin · Pull Request #4567 · pytorch/TensorRT

shoumikhin · 2026-08-23T14:12:37Z

The problem

Shipping the TensorRT delegate for ExecuTorch used to mean shipping ExecuTorch with it. The wheel
carried its own full copy of the ExecuTorch runtime, about 27 MB of it, built for one Python version.

That copy is the problem, not just the size. A user who already has ExecuTorch installed now has two
runtimes in one process. They can disagree, and when they do the failure is confusing: a program
loads, registers a backend against one runtime, and then runs against the other.

The change

The wheel now builds against the ExecuTorch you already installed, and ships three things: the
delegate library, a small Python package that registers it, and a CMake package so a C++ application
can link the prebuilt library.

The payload drops from about 27 MB to about 130 kB, and one wheel works on every supported Python
instead of one.

Getting it

One command. The executorch extra brings the delegate, ExecuTorch, Torch-TensorRT and PyTorch:

python -m pip install --pre "torch-tensorrt[executorch]" \
  --index-url https://download.pytorch.org/whl/nightly/cu132 \
  --extra-index-url https://pypi.org/simple \
  --extra-index-url https://pypi.nvidia.com

Swap cu132 for the CUDA version you run. Both extra indexes are needed, and so is --pre.
Measured in an empty environment:

package	version
torch-tensorrt	2.15.0.dev20260920+cu132
torch	2.15.0.dev20260920+cu132
torch-tensorrt-executorch-runtime	0.2.0.dev20260920+cu132
executorch	1.6.0.dev20260915+cu132
tensorrt	11.3.0.99
nvidia-cuda-runtime	13.2.86

ExecuTorch is older than the rest because the delegate pins the exact build it was compiled against,
which is what stops a silent mismatch.

On Arm the delegate needs glibc 2.35 where the others need 2.28, because its build container has no
devtoolset and the tag states that rather than claiming compatibility it does not have.

Using it from Python

Importing the package registers the delegate. There is nothing else to call:

from pathlib import Path

import torch
import torch_tensorrt_executorch_runtime  # noqa: F401
from executorch.runtime import Runtime

program = Runtime.get().load_program(Path("model.pte"))
forward = program.load_method("forward")
outputs = forward.execute((torch.ones((2, 3, 4, 4)),))

Using it from C++

Find the package by name and link it, the way ExecuTorch names its own backends:

find_package(executorch REQUIRED COMPONENTS backend_cuda kernels_optimized)
find_package(executorch_backend_tensorrt REQUIRED)

target_link_libraries(my_app PRIVATE
  executorch::runtime
  executorch::backend_cuda
  executorch::backend_tensorrt
  executorch::kernels_optimized
)

No header to include: the delegate registers itself when the library loads, and the rest is
ExecuTorch's own API.

using namespace executorch::extension;

Module module("model.pte");
auto input = make_tensor_ptr({2, 3, 4, 4}, data.data());
const auto outputs = module.forward(input);

Two behaviour changes worth knowing

Execution always waits. The runtime this plugs into has no asynchronous execute, so there was an
option to return early that could not be honoured safely, and it is gone.

A buffer on the wrong GPU is refused rather than written to. It used to be bound and then fail inside
TensorRT as an invalid program, which sends you to check a model that was fine.

…to fail A device id read out of the program file was accumulated in an int with no bound, so a value past the int range wrapped to a small plausible number and went straight to the call that selects a GPU. The digits come from a file, so the running total is kept in sixty four bits now and a value that will not fit is refused rather than truncated. Three tests could not fail, and two of them were mine. The check that refuses a buffer sitting on another GPU was read out of the whole source file, so a mention anywhere satisfied it. It now reads only the body that binds the buffers, and each of its four cases is required to turn that reading red, which is what stops the reading from passing against any source at all. It also says plainly what text cannot see: a check left in place and disabled by a condition around it. Only two cards catch that. The guard that rejects an absolute runpath entry could not see an empty one, which is the shape that makes a loader search the current directory. That case has a test now. Beyond those: a comment described a colon-split check no version of this repository ever had, two comments in the partitioner disagreed with the code under them, a probe docstring promised something it did not do, an alias for a removed name and a local import the file already had at the top were both dead, and the message for an aliased input with no address named the wrong thing. Three findings were declined rather than fixed, with the reasoning written down: a twenty comment sweep across thirteen files, forty nine redundant test markers whose removal changes nothing about collection and whose form fourteen neighbouring files already use, and a subclass whose only job would be to raise on a build flag no caller in the tree passes. One reported failure did not survive measurement. A type checker was said to be clean before this change and dirty after; the merge base already had errors of its own, and one of them survives the repair, so the headline was wrong in both directions. The mechanism was still worth three lines. Test plan: Parsed a blob whose device id is 4294967299. Before, it came back as 3 and was accepted. Now it is refused. A blob with an ordinary device id reads back unchanged, so the bound only catches what it should. Removed each of the four guarded behaviours from the real backend file in turn and confirmed the rewritten reading goes red for every one, then restored them. 543 passed in the lane that runs without PyTorch, 150 in the lane that needs it, and both delegate sources compile clean against the real ExecuTorch and TensorRT headers on an Arm device.

The runtime wheel required PyTorch and Torch-TensorRT, and it links neither. What it links is ExecuTorch, the inference library and the CUDA runtime, and those three are all it asks for now. Requiring Torch-TensorRT was the worse of the two, because this wheel is built from that project, so the requirement made it depend on its own parent. No resolver can satisfy that in general: it has to pick some build of this wheel, and an older published one requires a version of the parent that excludes the parent being built. The only thing that needed Torch-TensorRT was the deprecated forwarder, which imports it lazily and already says which package is missing when it is not there. Requiring an exact PyTorch was the wrong place for a real need. PyTorch is reached through ExecuTorch's Python bindings, not through anything here, and ExecuTorch deliberately leaves the choice of build to the user rather than pinning one. A wheel that plugs into ExecuTorch should not be stricter than ExecuTorch is. The reason given for both was that the exact version names the CUDA row every component has to share. That still matters, and it is still enforced: ExecuTorch is pinned with its label, and the inference library and CUDA runtime are pinned by version, so the row is named three times over by the things that actually carry it. Two tests asserted the old shape. Both now require ExecuTorch to keep its label and require the other two to be absent, one by reading the source and one by running the build and reading back what it declares. Test plan: Read what the delegate library declares it needs. It names libexecutorch, the ExecuTorch CUDA extension and thread pool, the inference library, the CUDA runtime and the C and C++ runtimes. Neither PyTorch nor Torch-TensorRT appears, and neither does any search path pointing at them. The registration module imports nothing outside the standard library and ExecuTorch. Made Torch-TensorRT unimportable and imported the wheel: it loads and registers. Made PyTorch unimportable as well and it fails, through ExecuTorch's bindings rather than through anything here, which is why PyTorch stays reachable but unpinned. Put each removed requirement back in turn and confirmed both tests go red, then removed them again. 543 passed in the lane that runs without PyTorch, 150 in the lane that needs it. One thing to know about the lockfile: every wheel of this already published still declares the old requirements, so a lock resolved against the current index keeps the cycle until a build carrying this change is published.

…rmatted The version this repository pins rewraps two lines the previous commits left, so the style job refused them. No code changed, and the parser behaves identically on all four of its cases afterwards.

The wheel checker walked five distributions and refused the wheel when any one was missing a pin, so removing the PyTorch and Torch-TensorRT requirements made the aarch64 build fail with a report that the wheel requires nothing at all. It walks the three the delegate links, and refuses the other two if they come back, because a checker that only looks for what it expects is how the old shape survived. Test plan: 543 passed in the lane that runs without PyTorch. The build lane is what actually exercises this script, so it is the signal to watch.

The build refuses a delegate whose C++ runtime version exceeds what the wheel tag promises, names the version, the symbols reaching for it and every disallowed version, and then stops. Correct, and it leaves a reader with no next step, which is how two attempts to build this wheel on ordinary machines stalled: the compiler on both was new enough to stamp a version the tag forbids. The message now says that, and that the remedy is the release container or an older compiler, and that there is deliberately no override, because a wheel that skips the check fails at load time on somebody else's machine instead of at build time on yours. Test plan: shell syntax clean, quotes balanced, the guard still refuses on no arguments and on an empty one, and 543 passed in the lane that runs without PyTorch.

Removing the PyTorch and Torch-TensorRT requirements left nine cases in the checker's test suite describing a wheel the build can no longer produce. Four asserted those requirements were present, two transformed a requirement that is no longer declared and so changed nothing, and three carried one of the removed names incidentally, which the new refusal caught before each case reached the branch it was written for. The two whose whole premise was the Torch-TensorRT requirement are gone; requiring the project that builds this wheel is now its own case and covers the same ground from the other side. The two that asserted absence became cases asserting presence is refused. One that relabelled a Torch-TensorRT requirement to prove a local label is rejected now relabels the inference library, which is a requirement that still exists and has the same rule. The refusal moved ahead of the version comparisons. A requirement that should not be there at all is a structural fault, and reporting a version detail about the requirements that should be there is the less useful answer when both are true. Several of those cases also hardcoded an inference library version, so on a machine carrying a different one they were refused for that instead of for their own reason. They read the installed version now, like the accepting case always did. One thing worth knowing for anyone who changes this checker. The whole group skips unless the installed ExecuTorch matches the repository pin, because the ExecuTorch comparison runs first and short-circuits every later check, so a mismatched environment refuses each wheel for the wrong reason. That is why this break reached CI with a green run behind it. The skip now says so. To run them, install the pinned ExecuTorch. Test plan: Ran the group with the pin temporarily aligned to the installed ExecuTorch, which is the only way to execute it on a machine that does not match: 36 passed, having been 4 failed and 32 passed before these fixtures were corrected. Restored the pin afterwards and confirmed the file is unchanged but for the intended edits. 543 passed in the lane that runs without PyTorch, 150 in the lane that needs it.

The checker reads each dependency version through the installed distributions, and its test already runs it with a directory of synthetic ones ahead of the real ones. Every dependency was listed there except ExecuTorch, so that one alone came from whatever the machine happened to have. Two consequences followed, and both cost real time. The whole group skipped unless the machine carried exactly the pinned ExecuTorch, because the ExecuTorch comparison runs first and short-circuits: with any other version installed, a wheel built to fail a later check is refused for ExecuTorch instead and the case proves nothing. So the group was invisible on most machines, which is how a change to the checker reached continuous integration with a green local run behind it. It was also invisible in a subtler way. A published build carries a label naming its CUDA row, and a local one may not, so the same fixture could pass on one machine and fail on the other for a reason that has nothing to do with what it tests. ExecuTorch is now a synthetic distribution like the rest, built from the repository pin with a label, so the checker sees a version this file chose. Both skips are gone and the group runs anywhere. That is 38 tests that used to skip and now run, and the change removes more lines than it adds. Several cases also hardcoded an inference library version, which refused them through somebody else's branch on a machine carrying a different one. They use the chosen version now. Test plan: The suite that needs PyTorch went from 150 passed with 62 skipped to 188 passed with 24 skipped, with no change to what any test asserts. Removed the checker's refusal of a requirement the delegate does not link, from the real file, and confirmed exactly the two cases covering it go red, then restored it. 543 passed in the lane that runs without PyTorch.

…ach other The check that refuses a buffer on the wrong GPU carried an exemption: if the two cards reported that they could reach each other, the buffer was allowed. That is wrong, and measuring it on two machines with four and two cards showed why. Every pair of cards on both machines reports that it can reach every other, while reaching actually requires the mapping to be turned on for a specific pair, and nothing in this stack turns it on. So the exemption permitted a pointer the engine cannot dereference, and the fault that follows is an illegal access that leaves the process unable to use the GPU at all. The base branch has no such exemption and refuses any buffer on another card, so this restores that rather than tightening anything. Enabling the mapping would make the permissive case genuinely work, and both measurements say so, but that is a process-wide change to the caller's GPU state and a delegate should not make it on the caller's behalf. It belongs in its own change, decided by whoever owns the process. The refusal now says this, because a user with two cards that report reachability will otherwise reasonably expect it to work. Two test assertions were pinning the exemption in place by requiring the capability call to be present. They require its absence now. Test plan: Measured on a four card machine and a two card machine independently. With the mapping left alone, a buffer on the second card faults with an illegal access. With the mapping turned on first, the same buffer runs correctly and returns exact values. Both agree, and both agree the other delegate in a coalesced program permits the same buffer silently rather than refusing it, so the two were never in conflict. Put the exemption back into the real file and confirmed the guard test goes red, then removed it again. The delegate compiles clean against the real runtime headers on an Arm device, 543 passed in the lane that runs without PyTorch and 188 in the lane that needs it. One correction to an earlier reading. The message naming a source and destination device in the other delegate's refusal reports memory kinds, not card numbers, and the identical message appears for a buffer on the correct card. That is what made it look as though the two delegates disagreed.

…ndings there are Damaging one character of a binding name in the program metadata did not refuse. The reader skips a key it does not recognise, which drops that whole binding, so a one input engine was set up as though it had none and the log said so: engine ready, zero inputs, one output. The failure came much later, at the first address the engine wanted and nobody had supplied, and the message there talks about supplying output buffers, which sends the reader to look at their own calling code rather than at a damaged file. The engine knows how many bindings it has, and that count was already being read a few hundred lines further down for a different purpose. Comparing it against the number the metadata names catches this at load, before anything is bound, and says which of the two disagrees. Test plan: Found by error-path testing on a two card machine, which damaged one byte of a name key and got a confident engine-ready line followed by a misattributed failure. The same testing confirmed the four harder corruptions of the engine bytes already refuse, through the inference library, then this delegate, then the runtime, and that reordering the metadata keys is bit identical. The delegate compiles clean against the real runtime headers on an Arm device, 543 passed in the lane that runs without PyTorch and 188 in the lane that needs it. Two defects found in the same testing belong to the runtime rather than here, and are recorded for report upstream rather than worked around. A null input pointer is refused on one of its two input paths and silently accepted on the other, returning a result identical to a correct run. And a genuinely full device turns a recoverable out of memory result into an abort, after naming the cause correctly one frame earlier.

…the document A program asking to run on a particular device ran somewhere else, silently. The reader searched the whole metadata document for each key and excluded the ranges the two arrays occupied. Those ranges are computed while walking, so one unrelated key shifted what they covered, the search matched a key nested inside that key, and the outermost object's own answer was never read. Which device it then used depended on what the nested value happened to be. On one document it read 0 instead of 9. On another it read a number with eight digits and handed that straight to the call that selects a device. All three keys read this way were affected, not just the device: the aliased input output list and the hardware compatibility flag went through the same search, so a document shaped to mislead one misled all three. Depth settles it. Every array in this document is nested inside the object, so a key at depth one is the object's own and a key inside an array never is. That also removes the two range arguments and the four lines that maintained them, and it fixes a second shape at the same time: a key whose string value happens to contain another key's name is no longer a match, because a name is only a name when a colon follows it. Test plan: Found by error-path testing on a four card machine, which loaded a real program with one extra metadata key and watched it run on the wrong device. Reproduced on two more machines by compiling the same reader against both versions and feeding it the same documents: two of five are read wrongly by the old one and correctly by the new one, and on a real program the old library called the device selector with a number no machine has while the new one reads the object's own value and runs. Three shapes behave identically either way, so the change is narrow: a tensor legitimately named after a key, a key whose value contains a key name, and an ordinary document with the keys in either order. Added both cases to the parser's own tests. That suite needs a test framework which is not installed on any device available here, so it compiles and runs in continuous integration rather than locally, and the behaviour it asserts is the behaviour measured above. 543 passed in the lane that runs without PyTorch, 188 in the lane that needs it, and both delegate sources compile clean against the real runtime headers on an Arm device.

…cument, and fit the messages in the log Three of the four metadata keys were changed to be found by depth rather than by searching the whole document. The bindings list was the fourth and was missed. It has the same failure as the others: a bindings list nested inside an unrelated key would be walked instead of the object's own, so every binding name would come from whatever that key held. It uses the same rule now, and no raw search remains in that file. Separately, three of this backend's error messages were longer than the runtime keeps. The runtime's log truncates at 256 bytes, and the longest of the three was 567, so the two thirds a reader most needed were never printed. In the launch failure that meant the reader was told about output addresses and never about the other two causes, one of which is the specific mistake of using the per-thread stream inside a green context. All three now name the cause and the remedy inside the first 256 bytes, most likely cause first. Test plan: Found by error-path testing on a four card machine, which read the remaining raw search out of the source and separately measured the truncation: 567 characters emitted, 256 kept, and the two phrases a reader needs both sitting in the 312 dropped. Measured every log string in the file against the limit, and none now exceeds it. Both delegate sources compile clean against real runtime headers on an Arm device. 543 passed in the lane that runs without PyTorch, 188 in the lane that needs it.

The rule for this repository is one concise line where a comment is needed at all, explaining why the code is the way it is. The comments this stack added did not follow it: twenty seven blocks across two files, not one of them a single line, the longest thirteen, and most of them narrating what had been tried before and what went wrong with it. That history belongs in the commit that made the change, where it already is, and a reader of the code has to walk past it every time. Each one is now the line that carries the reason, and the rest is gone. No code changed: eighty five comment lines removed, eighteen added. Test plan: both delegate sources compile clean against real runtime headers on an Arm device, 543 passed in the lane that runs without PyTorch and 188 in the lane that needs it.

The executorch extra asked for the delegate wheel by name. That wheel is published on the nightly channel only: the public index returns not found for it, and the released PyTorch index returns forbidden with no files on any of the three CUDA channels. So asking for the extra on any released build failed to resolve, which is a worse outcome than the second command the extra was meant to save. Every build of that wheel also carries a local version label, which the public index would refuse even if it were uploaded there. The extra now names only ExecuTorch, which is on every channel it will be resolved from, so it still does the useful half of its job. The delegate wheel is named on the install line instead, and the two documents that show the command say why. Two things follow from the same change. The lock file matches again, because what it records for both extras is what setup.py now declares; it never carried the delegate wheel, so a locked sync on any change to setup.py would have failed on the merge. And a comment justifying the version-free requirement is gone with it, which is just as well: it claimed the wheel pins this package exactly, and that pin was removed earlier today because nothing in the wheel links it. Test plan: Measured the two indexes directly rather than reasoning about them: not found on the public index, and forbidden with zero files on each released CUDA channel, against eighty files on the nightly channel. The test on this declaration asserted the wheel was present, so it now asserts the opposite, and the lock tests pass unchanged. 543 in the lane that runs without PyTorch, 188 in the lane that needs it. One thing this does not fix. Installing the wheel from an index has still never been done, because it is not published anywhere a person can reach. Both documents now show a command that was measured on the nightly channel, and that is the strongest claim available until the wheel ships.

The notes said running from more than one thread does not work, and gave counts: twenty one failures out of twenty one on one card, five out of five on another. Both halves are wrong as stated, and the error is in the pessimistic direction, so a reader was being told to avoid something that works. What actually decides it is whether the threads share one loaded program. One program per thread is clean: zero failures across every shape tried, at two threads and at four, with a barrier releasing them together or with staggered starts, sharing an input buffer, sharing a stream through the caller guard, and loading concurrently from cold. Twenty six thousand answers compared with no mismatch. Sharing one program across threads fails essentially always, and that is the runtime's own documented rule rather than something this backend could fix. The reason the old claim looked solid is that the measurement behind it could not have shown otherwise. It ran through Python, where the interpreter lock is held for the whole of a call, so two threads never overlap inside it and any result is a statement about a serialized loop. The overlap had to be established in C++ and confirmed with an in-flight counter before the question could be asked at all. The terminal symptom was also described as a hang. On one host it is a crash in a stream flush, and the hang and the crash are the same stack reached two ways, so the notes now say both. Test plan: Measured on a two-card host, one hundred calls per thread per attempt, twenty attempts per shape, with the threads confirmed to overlap rather than assumed to. The per-thread shape failed zero of twenty at both thread counts. The shared shape failed twenty of twenty at two threads and nineteen of twenty at four. Separately established that the Python path cannot produce an overlap at all, two independent ways, which is why an earlier clean result there proved nothing either way. Documentation only, no code changed. 227 passed in the lane that covers these notes and the guards around them.

… smaller repairs A review of this change found five things worth fixing in one of its four batches. The one that matters most was not the one it was filed as. The metadata readers for a number and for a boolean stopped as soon as they had something plausible and never looked at what came next. So a device written as 3.5 was read as device 3, 3e2 as 3, 12abc as 12, and 0x3 as device 0. A compatibility flag written as trueX read as true and falsehood read as false. Each of those is a wrong answer returned as success, from a file the reader is meant to be checking, which is the same shape as two other defects already fixed in this change. A value now has to end at a comma, a closing brace or bracket, or whitespace. The guard that checks the built library for symbols the loader must find accepted one that is hidden. A hidden symbol is present in the object and unreachable through the interface that looks for it, so the guard passed an artifact that fails at the first call. It checks visibility now. Two tests could not fail. Both filtered a list and then asserted something true of the empty result, so they passed whether or not the behaviour they name exists. They assert the filter found something first. The wheel checker described three packages as linked when only one of them is. And the example runner read a count with a function that reports no error, so a misspelled argument became zero runs and a silent success. Nine further findings in the batch were declined with reasons, and two were refuted with measurements, including one where the reviewer asked for a warning about a problem this change had already solved. Test plan: Every repair was measured before and after. The readers: six malformed values accepted with wrong numbers before, all six refused after. The symbol guard: purpose-built shared objects, one with a hidden symbol and one with it visible, refused and accepted respectively, where both were accepted before. The two tests were run against the code with the behaviour removed and go red. 543 passed in the lane that runs without PyTorch, 190 in the lane that needs it, and both formatters and the shell parser are clean at the versions continuous integration pins.

…ings a review found The wheel check compared the installed CUDA runtime against the version pinned in a file. That file names one row and the build matrix has three, so a correct build for either of the other two was rejected. It looked green only because the row the file names is the single Linux row today, so the first build for another row would have failed with the artifact being perfectly good. It compares the CUDA major now, which is what a row shares, and the exact version the wheel must require is unchanged, so nothing is loosened: that requirement is still pinned to the precise runtime the delegate was built against. Then six more, all found in the same batch. An import that can now fail was not named in the description, so somebody upgrading would meet it with no warning. A guard in the exporter turned a broken import into eleven quiet skips, which is the worst way to fail because the run still reports success. A note offering a way to skip a step said the result was redistributable, and it is not. A test named for a library the loader cannot find never ran any loader. A helper that asserts the device checks are present was written and never called from anything. A note said only integrated parts take the direct path, which is not what the code decides. A check for which side of a boundary is wrong searched the whole source rather than the message, so it passed on an unrelated match. And a marker meant for tests sat on a helper. Four findings were declined with reasons and one was refuted. Test plan: Each repair was measured rather than reasoned about. The wheel check was run against synthetic wheels for all three CUDA rows: one passed and two were rejected before, all three pass now, and a wheel requiring a runtime other than the installed one is still rejected. The tests that could not fail were run with the behaviour removed from the real files and go red. 547 passed in the lane that runs without PyTorch and 186 in the lane that needs it. Four tests moved from the second lane to the first because they no longer need PyTorch to run, so the total is unchanged.

…efactor broke Two more batches of a review of this change, decided in full: the real findings fixed, the rest refuted or declined with a reason. The forwarder in the companion package blamed the main package for being too old when it is simply absent, which is the ordinary state, because this package does not require it. It reads the module table to tell the two apart now, so a reader with none installed is told to install one rather than to upgrade something they do not have. The wheel check looked at three named requirements and could not see a fourth. An extra requirement was invisible to it, and an unlabelled one cleared the label check as well, so it refuses anything the delegate does not link. Two more tests could not fail. One asserted on a mutation that the base branch passes too. One parser test asserted a refusal that a valid file would also have produced. And a helper written to assert the device checks are present was never called from anywhere, while its assertions already exist a few lines away, so it is gone rather than wired up. One test was deleted rather than repaired. It read the forwarder source looking for a phrase, and that phrase moved into a helper when the message was improved. The test beside it drives the same two cases by running them, which is what the source search stood in for, so keeping both would pin the shape of the code rather than its behaviour. Test plan: Both formatters this repository gates on are clean at the pinned versions, and 187 passed in the lane that needs PyTorch. The lane that runs without PyTorch passed at 548 on the commit before this one. This machine is saturated by unrelated work, with a load average above one hundred, and a full run of that lane has not completed here since, so continuous integration is the check on it rather than a local result I can quote. Each repair was measured before and after by the batch that made it, including the mutation branch run against the base branch and the forwarder driven with the main package both absent and present.

An earlier commit in this branch took it out, on the grounds that the wheel is published on one channel and the extra would therefore fail to resolve elsewhere. That reasoning does not hold, and the commit is undone. The extra already resolves from a CUDA channel and not from the public index, because the ExecuTorch requirement beside it names a development build the public index does not carry: the floor is a dated 1.6 build and the public index stops at 1.5.0. So naming the delegate wheel costs nothing that was not already true of the extra, and it saves a reader finding a second command for themselves. The two wheels are also published together. Each CUDA channel carries both, built by the same matrix, so a build whose metadata names the delegate comes from a channel that has it. Test plan: Checked the premise rather than the conclusion. The extra floor is executorch>=1.6.0.dev20260915 and the newest executorch on the public index is 1.5.0, so the extra could not resolve there before this change either. Each nightly CUDA channel lists both wheels, eighty files of the delegate against seven hundred and eighty six of the main wheel on one of them, which is the same build matrix at two different retention windows. Also checked what a released build actually asks for, since that was the stated failure: the newest release that declares this extra names ExecuTorch alone, so no released build has ever tried to resolve the delegate. The test on this declaration expects both names again, and both install documents show the one command. 548 passed in the lane that runs without PyTorch, 187 in the lane that needs it.

Shortening that message to fit what the runtime keeps introduced a worse fault than the truncation it fixed. The new wording carried two placeholders for device numbers and the call passes one argument, so the log read two values off the stack and printed whatever was there. A caller hitting this case was told their buffer sits on a device that does not exist. The case is about whether an aliased input can be reached without staging, not about which device holds it, so there is one value to name and the message names it. Every log call in the file was then checked for the same mismatch, and none remains. Test plan: Compiled the file with the format check turned on, against the real runtime headers on an Arm device. The pushed version fails with the compiler naming the exact fault twice, that a conversion expects a matching argument. The corrected version compiles with no diagnostics at all. No job in continuous integration compiles this file with that check enabled, which is why the fault reached a pushed branch. Worth adding, and it belongs in its own change rather than here. 138 passed in the tests that read this message and the guards around it.

…the cards could reach each other An earlier commit in this branch refused any buffer on another card. That was too strict, and a review measured it: with the peer mapping enabled the inference library runs such a buffer correctly and returns the right numbers, and the refusal rejected it anyway. The commit before that one was too permissive in the other direction, allowing a buffer whenever the two cards reported they could reach each other, which is true of every pair measured and still faults, because the mapping has to be enabled for a specific pair. There is a third question, and it is the right one. The pointer attributes carry an address usable from the current device, or nothing when no such address exists. That is exactly what the engine needs to know. It answers nothing for a buffer on another card with no mapping, and an address once the mapping is enabled, so one test covers both directions without asking about capability at all. Test plan: Measured on a two-card host, with a kernel rather than by reading the flags. With the mapping off the attribute is empty, a kernel touching the pointer fails with an illegal access, and the check refuses. With the mapping enabled on the same pointer the attribute is set, the kernel writes the expected value, and the check allows. The previous behaviour allowed the first case, which is the fault it was added to prevent, and the current behaviour refused the second, which is the case this restores. The guard test asserts both halves now, the capability call absent and the usability test present, and goes red when the new test is removed from the real file. Compiles clean against real runtime headers on an Arm device, with the format check enabled. 551 passed in the lane that runs without PyTorch, 187 in the lane that needs it. Both two-card machines left the network before I could repeat the three cases myself, so the kernel measurement above is the one taken during the review round rather than a second independent run.

…hings a review found The two biggest test files in this change never ran on a pull request. The job that needs no PyTorch excluded them because they import it, and no other job picked them up, so the only lane running them was the nightly one. That is around a hundred and ninety tests covering the loader compatibility paths and the release matrix gates, none of which could fail a pull request. Three separate defects reached continuous integration today for exactly that reason. Both files read source, packaging metadata and workflow text rather than running a model, so a processor-only PyTorch is enough, and that is what the job installs now. Then five more, all from the same review. The guard that checks the built library lost the half that caught a second private copy of the C++ runtime, and another that caught a library defining the registration symbol instead of importing it. A delegate that defines it registers into its own registry, so the program loads and then reports no backend. Both halves are back, each with a test that fails when the guard is reverted. The build guard accepted any installed runtime whose local label merely began with the right two letters, never that the row matched, so a build against one CUDA row could ship claiming another. A green-context check turned a real setup failure into a successful skip, which is the same shape as several other faults found today: a run that could not test anything reported success. The deprecated forwarder told a reader with no Torch-TensorRT at all to upgrade one, and now names the case they are in and what to do about it. Its test checks both doors by driving them rather than by reading the source. Two findings were declined with reasons and two refuted with measurements. Test plan: 552 passed in the lane that runs without PyTorch, and 187 in the lane that needs it. The new job step was exercised by running exactly its command list against a processor-only PyTorch. Every guard repair was measured with purpose-built shared objects, one for each thing the guard should accept and one for each it should refuse, and each new test goes red when the behaviour is removed from the real file rather than from a copy.

The Python formatter this repository gates on wanted one file reformatted, so the lint job failed on an otherwise green head. My mistake: the previous commit ran the formatter over the files I had edited by hand and not over the one that arrived in a patch. Swept every Python file this stack touches afterwards, and this was the only one. Test plan: the formatter reports no changes wanted across the stack, and the tests in that file pass.

…clares it nowhere Removing PyTorch from this wheel's requirements was half right and shipped broken. The delegate really does link no PyTorch, and pinning one really would be this wheel deciding something ExecuTorch leaves to whoever installs it. But ExecuTorch's own Python imports PyTorch at the top of a module the delegate loads through, and nothing in the remaining dependency set asks for it: ExecuTorch names only one other package, and neither that one nor the tokenizer beside it mentions PyTorch. So an install of this wheel plus ExecuTorch resolves cleanly and then fails on the first import. It is required by name now, with no version bound. Present, so the install works. Unbounded, so the choice stays where ExecuTorch put it. The wheel check follows: PyTorch moves off the forbidden list and onto a short list of names that must be present and must carry no bound, so a future pin is refused rather than shipped. Only the parent project stays forbidden outright, since requiring it would make this wheel depend on the thing that builds it. Test plan: Reproduced the break on an Arm device. Copied a working environment, uninstalled PyTorch, left everything else in place, and imported the package: it fails with the module not found, reported as a compatibility error. Confirmed from the installed metadata that the import really is at module level and that no remaining dependency declares PyTorch. Two independent runs reached the same conclusion, one by uninstalling and importing, one by reading the import and the metadata. The check's own test goes red when the new rule is removed from the real file. A fixture that pins PyTorch is refused for the pin and names it; the fixtures that declare it unbounded are accepted. 552 passed in the lane that runs without PyTorch, 187 in the lane that needs it. Also here: an export script wrote over its target before its own validity checks ran, so a rejected export destroyed a good program and exited non-zero, leaving neither. Measured on the same device, target hash changed with exit 1. It writes beside the target and moves it into place only after every check passes.

…could not fail Three findings from a review, each measured before and after. The copy that reflects an in-place update back to its output asserted device to device. That is right when both sides are device memory and wrong when either is not, and on a device that stages rather than binds either side can be host memory. Measured on three machines: the asserted form returns correct bytes on a discrete card and on the larger integrated part, and fails on the smaller one, where letting the runtime work the direction out measured clean. One enum, and the message no longer names a direction it does not know. Then two checks that passed whatever the code did. One pins the build matrix filter. Every row the surrounding rules already keep is a CUDA 13 row, so the one flag this check is about is the only thing that can drop the older row, and with that flag deleted the old check still passed. The new assertions fail on exactly that deletion. The other pins the shipped CMake package finding its own copy of the delegate. A library left in a parent directory used to satisfy it, which is the failure the check exists to prevent, so a consumer is now configured against a decoy in the parent and the configure is required to fail, then required to succeed against the package's own copy. A proposal to define the backend identifier once and import it everywhere was declined. The five spellings live in three separately shipped artifacts, so a single source needs either generated code or an import across wheel boundaries, and that is a layer this change should not grow by. A check pins the native spelling against the name the exporter derives instead, so a rename turns it red rather than producing programs nothing can load. Test plan: Each check was proved by deleting the thing it names from the real file on disk and watching it go red, then restoring it. The copy direction was measured on three devices, two integrated and one discrete. 553 passed in the lane that runs without PyTorch, 187 in the lane that needs it, and the delegate compiles clean against real runtime headers on an Arm device with the format check enabled.

…ment A review of the minor findings turned up one that outranks its label. The check that the delegate keeps the search path it needs read the whole build file, comments included, so commenting the setting out left the check green while the built library lost the path. That is the eighth check in this change found to pass whatever the code did, and this one guards how the library finds the libraries it links. It reads the file with comment lines removed now, and goes red when the setting is commented out. The rest of that batch was small and is grouped here because the reasons repeat. A handful of comments described code that has since changed, and each is now a shorter correct line. Two spellings of one loader disagreed across three places, which would send a reader to the wrong one. An example refused an argument it should accept and accepted one it should refuse. A helper in the pin updater and an unused import went, being dead. Most of the batch was refused rather than fixed, and the reasons are worth recording. Anything cosmetic whose repair touches code that tests read was declined, because a batch of individually correct naming edits was tried earlier in this change and broke eleven of them. Every proposal to introduce a shared constant or a helper was declined too: the change is already large, and the identifiers those proposals would unify are spelled in separately shipped artifacts, so one source needs generated code or an import across packaging boundaries. Test plan: The search path check was proved by commenting the setting out in the real build file and watching it go red, then restoring it. Before that edit the same mutation left it passing. 553 passed in the lane that runs without PyTorch, 187 in the lane that needs it, and both formatters are clean at the versions continuous integration pins.

…things The companion's build recipe exported one CUDA row and then installed from another and labelled the wheel with the second, so a reader following it built against one set of headers and published a wheel claiming a different row. The two uses read the export now, so a reader edits one line and the whole recipe follows. The check on that recipe expands the variable before parsing the version, and refuses anything still unexpanded, so the recipe cannot go back to disagreeing with itself quietly. The reference runner's install command named one index and needed two. Without the inference library's own index, pip finds only a source distribution and spends about twenty minutes failing to build it, which looks like a broken instruction rather than a missing index. Two comments said things the code does not do. One claimed inputs always come from device memory when that holds only with a flag, and one claimed the module interface backs device-tagged arenas with device memory. An unused include went with them. Test plan: The recipe check was proved against the recipe as it stood: the version it exported could not be parsed at all, which is how the disagreement survived. It parses now, and a deliberately unexpanded value is refused. 553 passed in the lane that runs without PyTorch, 187 in the lane that needs it, and both formatters are clean at the pinned versions. One note for whoever reads this next to the commit before it. Two review batches proposed different repairs for the recipe: align the exports downward to match the uses, or have the uses follow the export. The second is here, because it leaves one line to edit rather than three to keep in step.

The last of four batches of minor and nit findings. Nine were worth doing, sixteen were not, and the reasons for refusing are as useful to record as the fixes. Worth doing, grouped by what they were: Two more checks that could not fail, which makes ten found in this change. One asserted on a value it never compared, and one accepted an empty result as agreement. Both go red now when the behaviour they name is removed. A wheel check accepted a requirement list with a name it had no opinion about, so an unexpected requirement passed silently. It refuses anything outside the set it knows. A job in the delegate's own workflow ran on one Python version where the matrix has several, so the other rows were never exercised by it. Three notes described behaviour the code no longer has, and one paragraph of the companion's notes repeated what the paragraph above it already said. Each is shorter and correct now, and the repeated one is gone rather than reworded. Refused, with the reasons: Every proposal to rename something, to add a helper, or to share a constant across files. The change is already large, its size is a review complaint of its own, and a batch of individually correct renames was tried earlier here and broke eleven tests. A name somebody would have chosen differently is not a defect. Every request to add explanatory commentary. This repository asks for one concise line where a comment is needed at all, explaining why rather than what, and ninety such lines were removed from this branch earlier for breaking that. Test plan: Each new check was proved by removing from the real file the thing it names, running it, watching it go red, and restoring it. 554 passed in the lane that runs without PyTorch and 188 in the lane that needs it, each one more than before this change, and both formatters plus the workflow parser are clean.

…laming the wrong thing End to end testing on four machines found one misuse nothing reports and two messages that send a reader to the wrong place. Freeing an input from another thread while the call runs returns success with a wrong answer. Measured on the device that hands a caller's pointer straight to the engine: 253 of 300 runs came back with the right rank, the right type, every value finite, and a status of success. The model there is linear, so the recycled bytes produce a scaled version of the right answer, which is the shape a reader would never question. Nothing below this layer can catch it, because the delegate is handed an address and by the time the device reads it the memory belongs to somebody else. So the rule belongs to the caller, and it was written nowhere. It is in the header and in the notes now, with the measurement and with the reason the Python path cannot reach it: a tensor there owns its storage. Then two messages. A failure reported when the call waits for the device to finish was classified as an invalid program, which sent a reader to re-export a program whose own consistency was already checked at load. The fault is asynchronous and this layer cannot attribute it, so it is reported as an internal failure now and the message names the three usual causes in order rather than picking one. A failure copying an output out asserted that no buffer had been supplied for it. That is usually right and was wrong in the case that mattered: in a program split across two backends, a fault in the other one leaves the device unusable and every later call fails the same way, so the reader was sent to their own output handling while the real error sat five messages earlier. It names both possibilities now and says to read the first error. Test plan: The lifetime case was measured three hundred times on hardware, and the zero-copy property it depends on was confirmed two ways: by counting the calls inside execute, and by making every allocation fail after load, where execute still returned the right answer because it allocates nothing. The misattributed synchronise failure was reproduced with an input buffer shorter than its shape. The output message was reproduced in a two-backend program with an unmapped buffer on another card, where six errors arrive and only the first is honest. Every message in the file is inside the length the runtime keeps. 554 passed in the lane that runs without PyTorch, 188 in the lane that needs it, and the delegate compiles clean against real runtime headers on an Arm device with the format check enabled.

…er buffer is The ordinary output path asks which device a caller's buffer is on and refuses one this engine cannot reach. The aliased path returns before it gets there, so it never asked. It checked the buffer had an address and then copied into it, whatever card it was on. Two independent runs found this by reading, and a third confirmed by tracing the branch: the aliased output is the destination of a copy the engine issues, with no residency check anywhere above it. It gets the same check now, with the same message shape as the ordinary one. Test plan: The copy that reflects an in-place update was measured on the device that stages rather than binds, with a control built from the previous code so the difference is visible rather than argued: aliased input aliased output this version previous version device device 5 runs, 0 failures 2 runs, 0 failures device host 5 runs, 0 failures 2 runs, 2 failures managed host 5 runs, 0 failures 2 runs, 2 failures So the direction change committed earlier is confirmed on exactly the pair it was made for, and the control fails there with an invalid argument from the copy, which is what a reader used to see. The new residency check is pinned by an existing guard with a new case, and that case goes red when the check is deleted from the real file. 555 passed in the lane that runs without PyTorch, 188 in the lane that needs it, and the delegate compiles clean against real runtime headers on an Arm device with the format check on.

Asking for PyTorch by name with no version is deliberate: ExecuTorch imports it and leaves the choice to whoever installs it. Measured on a two card machine, the cost of that is worth writing down. Installing this wheel on its own, with no index and nothing else named, resolved PyTorch 2.10, which is older than the ExecuTorch build this wheel pins. That combination installs, imports, and loads a program, all reporting success, and then fails at the first run with a message naming a tensor and saying nothing about versions. Neither a pin nor a floor belongs here. The delegate links no PyTorch, and the version it needs is whatever the ExecuTorch it pins was built against, which ExecuTorch does not declare. Guessing on ExecuTorch's behalf would be this wheel deciding something it cannot know. So the install section says what the failure looks like and that following the documented command avoids it. Test plan: Both halves measured on the same box within minutes of each other: the documented install ran the same program against eager with no difference at all, and the bare install failed at the first run as described. One caveat recorded rather than hidden: the two environments also differed in Python version, so that single measurement does not prove the PyTorch version was the only cause. 555 passed in the lane that runs without PyTorch, 188 in the lane that needs it.

meta-cla Bot added the cla signed label Aug 23, 2026

github-actions Bot added component: tests Issues re: Tests component: build system Issues re: Build system component: api [Python] Issues re: Python API component: api [C++] Issues re: C++ API labels Aug 23, 2026

shoumikhin mentioned this pull request Aug 23, 2026

Track the ExecuTorch pin and move it to new nightlies automatically #4566

Merged

github-actions Bot requested a review from narendasan August 23, 2026 14:13

shoumikhin force-pushed the executorch-slim-runtime-wheel branch 3 times, most recently from 44796ff to 3c104cb Compare August 23, 2026 19:00

shoumikhin marked this pull request as ready for review August 23, 2026 19:08

shoumikhin force-pushed the executorch-slim-runtime-wheel branch from 3c104cb to 4adc20b Compare August 23, 2026 19:27

github-actions Bot added the documentation Improvements or additions to documentation label Aug 23, 2026

shoumikhin force-pushed the executorch-slim-runtime-wheel branch from 4adc20b to 7cac1af Compare August 23, 2026 19:30

shoumikhin marked this pull request as draft August 23, 2026 19:31

shoumikhin marked this pull request as ready for review August 23, 2026 19:57

shoumikhin force-pushed the executorch-slim-runtime-wheel branch 7 times, most recently from ff3379e to a008221 Compare August 24, 2026 17:00

lanluo-nvidia added this to the v2.15.0 milestone Aug 24, 2026

lanluo-nvidia added the ci: nightly Run the nightly lane (all tiers incl. llm / kernels / distributed) on every push label Aug 24, 2026

github-actions Bot requested a review from lanluo-nvidia August 24, 2026 17:03

shoumikhin force-pushed the executorch-slim-runtime-wheel branch 4 times, most recently from 49b052e to 370c368 Compare August 25, 2026 06:55

pytorchbot added 30 commits September 19, 2026 21:41

style: run the pinned clang-format over the two files this stack refo…

2e9858c

…rmatted The version this repository pins rewraps two lines the previous commits left, so the style job refused them. No code changed, and the parser behaves identically on all four of its cases afterwards.

Navigation Menu

Ship only the TensorRT delegate in the ExecuTorch runtime wheel - #4567

Ship only the TensorRT delegate in the ExecuTorch runtime wheel#4567
lanluo-nvidia merged 184 commits into
pytorch:mainfrom
shoumikhin:executorch-slim-runtime-wheel

shoumikhin commented Aug 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shoumikhin commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

The change

Getting it

Using it from Python

Using it from C++

Two behaviour changes worth knowing

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shoumikhin commented Aug 23, 2026 •

edited

Loading