From 32080a4844a8c91e052cabdd8d28f81ac4d39bc5 Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Fri, 7 Aug 2026 17:28:39 +0800 Subject: [PATCH 1/8] Add dynamic shape explainer --- dynamic-shape-explainer.md | 361 +++++++++++++++++++++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100644 dynamic-shape-explainer.md diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md new file mode 100644 index 00000000..b2fcc826 --- /dev/null +++ b/dynamic-shape-explainer.md @@ -0,0 +1,361 @@ +# Dynamic Shape Explainer + +## Authors +- [Bin Miao](mailto:bin.miao@intel.com), [Wanming Lin](mailto:wanming.lin@intel.com) (Intel) + +## Participate +- [Support flexible input sizes #883](https://github.com/webmachinelearning/webnn/issues/883) + +## Table of contents +1. [Introduction](#introduction) +2. [Goals](#goals) +3. [Non-goals](#non-goals) +4. [Use Cases](#non-goals) +5. [Proposed API](#proposed-api) +6. [Design Discussion](#design-discussion) +7. [Considered Alternatives](#considered-alternatives) +8. [Privacy & Security Considerations](#privacy--security-considerations) +9. [Future Consideration](#future-consideration) +10. [Open Questions](#fine-grained-shape-queries) +11. [References & Acknowledgements](#references--acknowledgements) + +## Introduction +Currently, all `MLOperand` instances within a WebNN graph are constrained to static shapes. Dimensions must be fully resolved during the [MLGraphBuilder.build()](https://www.w3.org/TR/webnn/#api-mlgraphbuilder-build) compilation phase. This constraint significantly limits the API's adaptability for modern machine learning workloads, where input dimensions often remain indeterminate until the point of inference: + +- **Transformer / LLM Decoding:** These workloads involve iterative execution of identical compute graphs with varying sequence lengths. Static shaping necessitates graph recompilation for each distinct sequence length, incurring prohibitive latency. + +- **Vision Encoders:** These architectures often process inputs of arbitrary resolutions, causing spatial dimensions to fluctuate dynamically throughout the network. + +- **Generative Image Models:** These models frequently compute intermediate tensor shapes at runtime (e.g., dynamic padding to block-size alignments), where padding requirements are strictly dependent on input dimension variance. + +Existing framework workarounds — such as frequent graph recompilation or falling back to slower, CPU-based execution environments (e.g., ONNX Runtime Web via WebAssembly) — introduce significant performance bottlenecks, particularly in latency-sensitive autoregressive decoding tasks. + +This proposal introduces **Dynamic Shape** for WebNN, enabling graph dimensions to remain unresolved during compilation and instead derive concrete values from input tensors at dispatch time. This mechanism permits a single compiled graph to accommodate a diverse range of runtime input sizes. This document details the design explored in the Chromium implementation and serves as a technical reference for the Working Group's resolution of [Issue #883, "Support flexible input sizes"](https://github.com/webmachinelearning/webnn/issues/883). + +## Goals +- Allow a single compiled `MLGraph` to execute across varying runtime input sizes, without rebuilding. + +- Model dynamism the way the underlying runtimes already do: a dimension is either a **static size**, a **named dynamic** dimension (a symbolic name), or an **unnamed dynamic** dimension (fully unconstrained). + +- Defer shape validation that cannot be decided at build time to the point where concrete input shapes are known, without weakening the checks that can be decided at build time. + +- Let a framework learn a graph's concrete output shapes for a given set of input shapes **before** dispatching it, so it can allocate output tensors of the right size. + +- Support models whose intermediate shapes are computed at runtime from other shapes (e.g. `padded_len = seq_len + (-seq_len % block)`), not just models that thread an input dimension straight through. + +## Non-goals +- **We do not resolve shapes from input tensor data.** Dynamism that depends on the *values* inside a tensor is out of scope. We resolve only *Symbolic sizes*: dimensions derivable by algebra from the input shapes and build-time constants. + +- **We do not redefine operator semantics.** The operators added here mirror the semantics of their existing static counterparts; they only move shape parameters from build-time attributes to runtime operands. + +## Use Cases + +### Variable sequence length +A web app runs an SLM decoder. The sequence length grows by one every step, so with static shapes the graph must be rebuilt (or a family of graphs pre-built) for each length. + +```js +// Before: the sequence length is baked into the graph. +const input = builder.input('attention_mask', {dataType: 'float32', shape: [1, 128]}); +// ...a different compiled graph is required for each sequence length. +``` + +With dynamic shapes, the sequence length is declared as a named dynamic dimension and the graph is compiled once: + +```js +// After: 'sequence_length' is a named dynamic dimension. +const input = builder.input('attention_mask', {dataType: 'float32', shape: [1, 'sequence_length']}); +const graph = await builder.build({output}); + +// The same graph runs at any sequence length. +mlContext.dispatch(graph, {'attention_mask': tensorWithLen(1)}, {'output': out1}); +mlContext.dispatch(graph, {'attention_mask': tensorWithLen(37)}, {'output': out2}); +``` + +Every operand whose shape depends on `sequence_length` carries the dynamism through the graph automatically. + +### Shapes computed at runtime +Some models contain genuine shape arithmetic that must run at inference time. For example, a "pad the sequence up to a multiple of the block size" step computes the padding amount from the (dynamic) sequence length: + +```js +padding_needed = (-seq_len) mod block // e.g. block = 32 +padded = pad(x, /*end=*/[padding_needed, 0]) // now a multiple of 32 +``` + +Expressing this requires reading a shape *as data*, doing arithmetic on it, and feeding the result back into an operator as a shape parameter: + +```js +const shape = builder.shape(x); // uint32 1-D tensor [seq_len, 2560] +const seqLen = builder.slice(shape, [0], [1]); // [seq_len] + +// (-seq_len) mod 32 +const pad = builder.modulusFloor(builder.neg(seqLen), builder.constant(..., [32])); +const zero = builder.constant(..., [0]); // no padding on the trailing axis +const padded = builder.padDynamic(x, builder.constant(..., [0, 0]), builder.concat([pad, zero], 0)); +``` + +This motivates the **shape-as-data operators** described below. + +### Getting output shapes before dispatch +A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs (it is derived inside the subgraph). To allocate the output tensor, the framework needs the concrete output shape before it dispatches: + +```js +const outShapes = await graph.computeShapes({'attention_mask': [1, 37]}); +// => {'output': [1, 74]} + +// Allocate output MLTensors of the resolved size, then dispatch. +``` + +## Proposed API +The web-facing surface changes fall into three groups: 1) the dimension model and descriptors, 2) new `computeShapes()` methods and 3) new operators. The runtime behavior behind these — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape folding** — is described under [Design Discussion](#design-discussion). + +### 1. Dimension model and descriptors +A graph input's dimension may now be a number (static integer), a string (a **named dynamic** dimension mapped to a runtime symbolic name), or — as a `null` element of the shape sequence — an **unnamed dynamic** dimension. This is expressed with a new `MLDimension` type and a new `MLInputOperandDescriptor`: + +```webidl +// A single input dimension: a number is static; a string is a named dynamic +// dimension; a null element (below) is an unnamed dynamic dimension. +typedef ([EnforceRange] unsigned long or DOMString) MLDimension; + +dictionary MLInputOperandDescriptor { + required MLOperandDataType dataType; + required sequence shape; +}; +``` + +Reading a shape back, `MLOperand.shape` is widened accordingly: its elements may now be strings, an unnamed dynamic dimension surfaces as `"?"` (a provisional representation — see [Open Questions](#fine-grained-shape-queries)), and the whole attribute is `null` for an **unranked** operand — one whose rank is not yet known (see [Dimension semantics](#dimension-semantics)): + +```webidl +interface MLOperand { + readonly attribute MLOperandDataType dataType; + // Was: FrozenArray shape; + readonly attribute FrozenArray? shape; +}; +``` + +The plain `MLOperandDescriptor` is deliberately **unchanged** (static-only): it describes `constant()` data, and `computeShapes()` likewise returns fully concrete (static) shapes. Dynamism thus lives only on graph *inputs* and propagates from there. + +```webidl +// Unchanged — always concrete. +dictionary MLOperandDescriptor { + required MLOperandDataType dataType; + required sequence<[EnforceRange] unsigned long> shape; +}; +``` + +### 2. `computeShapes()` +`computeShapes()` runs the same shape inference and folding as dispatch, but early and without executing the graph, returning the concrete output shape for each output given concrete input shapes: + +```webidl +partial interface MLGraph { + record> computeShapes( + record> inputShapes); +}; +``` + +The programming model becomes **build → (optionally) computeShapes → dispatch**. + +This method enables frameworks to determine the output tensor sizes for dynamic subgraphs whose shapes are difficult to infer. Exposing it reduces redundant computation overhead. + +### 3. Shape-as-data operators +Threading a dynamic input dimension through the graph is not sufficient on its own; real models compute with shapes. This proposal adds a family of operators that treat a shape as a runtime tensor, plus dynamic variants of existing operators that take their shape parameters as **operands** rather than build-time attributes. + +```webidl +partial interface MLGraphBuilder { + // Read an operand's shape as a runtime uint32 1-D tensor. + MLOperand shape(MLOperand input, optional MLOperatorOptions options = {}); + + // Shape generators / arithmetic on shape tensors. + MLOperand range(MLOperand start, MLOperand limit, MLOperand delta, + optional MLOperatorOptions options = {}); + MLOperand modulusFloor(MLOperand a, MLOperand b, optional MLOperatorOptions options = {}); + MLOperand modulusTruncate(MLOperand a, MLOperand b, optional MLOperatorOptions options = {}); + + // Rank-changing operators (the seam where dynamic rank originates). + MLOperand squeeze(MLOperand input, optional MLSqueezeOptions options = {}); + MLOperand unsqueeze(MLOperand input, sequence<[EnforceRange] unsigned long> axes, + optional MLOperatorOptions options = {}); + MLOperand reshapeTo2d(MLOperand input, optional MLReshapeTo2dOptions options = {[EnforceRange] unsigned long axis = 1;}); + + // Dynamic variants: shape parameters are operands, evaluated at dispatch. + MLOperand reshapeDynamic(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {}); + MLOperand expandDynamic(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {}); + MLOperand sliceDynamic(MLOperand input, MLOperand starts, MLOperand sizes, optional MLSliceDynamicOptions options = {}); + MLOperand padDynamic(MLOperand input, MLOperand beginningPadding, MLOperand endingPadding, optional MLOperatorOptions options = {}); + + // Equal-split (scalar count) and explicit-splits (operand) forms. + sequence splitDynamic(MLOperand input, [EnforceRange] unsigned long splits, optional MLSplitOptions options = {}); + sequence splitDynamic(MLOperand input, MLOperand splits, optional MLSplitOptions options = {}); + MLOperand resample2dDynamic(MLOperand input, optional MLResample2dDynamicOptions options = {MLOperand sizes, ...}); + MLOperand tileDynamic(MLOperand input, MLOperand repetitions, optional MLOperatorOptions options = {}); +}; +``` + +Three notes on the design of this family, rather than a per-operator walkthrough: + +- **Each `*Dynamic` operator is a full dynamic mirror of its static counterpart.** `sliceDynamic` uses the same `starts + sizes (+ strides)` semantics as static `slice`; splitDynamic mirrors split's explicit and equal-split forms and its `axis` option. Keeping them one-to-one lets the static operators stay static-input-only and avoids divergent validation. + +- **`squeeze` / `unsqueeze` / `reshapeTo2d` are the origin of dynamic rank.** E.g. a no-axes `squeeze` removes every size-1 dimension, so its output rank depends on runtime data. Emulating this logic requires complex subgraph chains to calculate runtime shapes, resulting in significant graph bloat and excessive shape inference overhead. Native operator support provides a more efficient and direct representation of these rank-changing transformations. + +- **We do not adopt ONNX's -1 (auto-infer) / 0 (copy) reshape conventions.** These are framework-level conveniences that a framework can lower into a shape subgraph before calling WebNN, and not all runtimes support them; keeping them out of WebNN avoids baking one framework's convention into the platform. So frameworks must instead use a subgraph chain to calculate these dimensions dynamically: + +For example, when the target shape is a runtime operand (values unknown at build time), the framework resolves the dimensions element-wise across the entire shape vector: + +1. **Resolve 0:** `equal(targetShape, 0)` → `where(mask, shape(input), targetShape)` → `shapeNoZero`. +2. **Resolve -1:** + - `total = reduceProduct(shape(input))` + - `known = reduceProduct(where(equal(shapeNoZero, -1), 1, shapeNoZero))` + - `inferred = div(total, known)` +3. **Final Assembly:** `where(isNeg1, inferred, shapeNoZero)` → `cast(uint32)` → `reshapeDynamic`. + +*(Optimization: This chain may be skippable if the framework can prove the operand is already sentinel-free.)* + +## Design Discussion +This section describes the runtime behavior that gives it meaning — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape folding**, plus the dimension semantics they honor. + +### Dimension semantics +- **Named dimensions are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"). An unnamed dimension carries no such constraint, and a dynamic dimension has **no min/max bound** — anything not provably static simply defers. + +- **Derived dimensions are never given a synthesized name.** A named dimension survives only on a 1:1 pass-through; any dimension *computed* from a dynamic one becomes unnamed. This avoids inventing symbolic identities the runtime cannot honor. + +- **Unranked operands.** For example, a no-axes `squeeze` removes every size-1 dimension, so its output rank depends on runtime data — the operand is unranked (its `MLOperand.shape` is null) until `computeShapes()`/`dispatch()` recovers it. + +### Deferred validation +Validation splits across two phases: + +- **At build time**, we validate only what is knowable without concrete shapes: data-type compatibility, rank constraints (e.g. `conv2d` needs rank 4), same-name symbolic consistency, and any **definite static contradiction** (e.g. reshaping a static `[2, 3]` to `[7]`). + +- **At dispatch time** (and at `computeShapes()`), once concrete input shapes are known, we run **shape inference** — a forward propagation of each operand's concrete *shape* — over the whole graph, and validate the resulting concrete shapes against every constraint, including buffer sizes. + +Deferring this work is the inherent trade-off of dynamic shapes: the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so a user agent should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. + +The build-time checks are expressed as **three-valued** dimension predicates. Instead of "equal / not-equal", a comparison is *provably-equal*, *provably-unequal*, or *unknown (defer)*. Only a provable contradiction is rejected at build time: + +```cpp +// Reject only when BOTH dims are static AND differ; otherwise defer to dispatch. +bool DimensionsAreDefinitelyUnequal(Dimension a, Dimension b) { + if (IsStatic(a) && IsStatic(b)) + return StaticValue(a) != StaticValue(b); + return false; // unknown -> defer +} +``` + +The same predicate is applied uniformly across operators that impose cross-dimension constraints, e.g. `matmul`'s contraction dimension, concat's non-concatenated axes, `reshape`'s element-count product, broadcasting, and so on. + +### Shape folding at dispatch +*Shape folding* is the dispatch-time evaluation of a `shape()`-rooted chain down to the concrete values a shape parameter needs. It is deliberately narrow — the shape-calculation operations only: arithmetic and structural transforms on shape tensors (usually integer, sometimes float, e.g. reciprocal) — and never the data-producing operators such as `conv2d` or `matmul`, which do not sit on a shape chain. It is the value-computing counterpart to *shape inference* ([Deferred validation](#deferred-validation)), which propagates operand shapes across the whole graph. + +A small chain makes this concrete. To reshape a `[1, 'seqlen', 512]` tensor into `[1, 'seqlen', 8, 64]` (splitting the static hidden size into 8 heads × 64) while keeping the dynamic sequence length: + +```js +const s = builder.shape(x); // uint32 1-D: the runtime dims of x +const batchSeq = builder.slice(s, [0], [2]); // first two dims → [1, seqlen] +const heads = builder.constant(/*uint32*/ ..., [8, 64]); // static tail +const newShape = builder.concat([batchSeq, heads], 0); // [1, seqlen, 8, 64] +const y = builder.reshapeDynamic(x, newShape); // y: [1, 'seqlen', 8, 64] +``` + +At dispatch with `seqlen = 37`, folding walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That folded value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. + +What folding may read is bounded by the root: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#fine-grained-shape-queries) case. + +### Backends mapping + +#### ORT +- **Named dynamic dimension** → ONNX symbolic name ([OrtApi::SetDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a6575872736b924b47a382deb97e2fc17)): + +```cpp +WebNN: shape=['batch', 512] +ONNX: shape=[dim_param:'batch', 512] +``` + +- **Unnamed dynamic dimension** → ONNX free dimension ([OrtApi::SetSymbolicDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#aa5b1654064d833a515f3acfcdcc5e81d)): + +```cpp +WebNN: shape=[null, 512] +ONNX: shape=[-1, 512] +``` + +#### LiteRT +- **Named/Unnamed dynamic dimension** → LiteRT unknown dimensions: + +```cpp +WebNN: shape=["batch", null, 512] +LiteRT: shape=[-1, -1, 512] +``` + +Represented in the LiteRT flatbuffer schema as: + +```cpp +const flatbuffers::Offset> dimensions +``` + +#### Core ML +- **Named/Unnamed dynamic dimension** → Core ML unbounded ranges dimensions: + +```cpp +WebNN: + shape=["batch", null, 512] +``` + +```cpp +// Core ML: +auto* size_range = shape_range->add_sizeranges(); +size_range->set_lowerbound(1); +// set upper bound to maximum long value +size_range->set_upperbound(std::numeric_limits::max()); +``` + +## Considered Alternatives + +### Per-operator build-time loosening +The first cut loosened each operator's build-time validator independently to tolerate dynamic dimensions. This scattered subtly different "is this okay?" logic across dozens of operators. We unified it behind the three-valued dimension predicates in [Deferred validation](#deferred-validation), so every operator defers the same definition of "unknown" and only rejects provable contradictions. + +### Bespoke dynamic operators +We considered giving each `*Dynamic` operator a shape signature tailored to its most common use, rather than mirroring the static operator. We rejected it for the reasons in [Shape-as-data operators](#3-shape-as-data-operators): faithful mirroring keeps the mental model small and lets the static operators stay static-input-only. + +## Privacy & Security Considerations +Dynamic shapes add no new fingerprinting or cross-origin surface: they expose no device or environment information a static graph does not, and folding and inference read only shapes the page itself supplied. The considerations below are therefore about security. + +The renderer is untrusted, so the service must remain safe on any graph a compromised renderer can construct, including ill-formed dynamic graphs: + +- **No dereference of an absent rank.** Unranked operands (e.g. from a no-axes squeeze) are handled uniformly by each shared validator — propagate, resolve, or cleanly reject — and unranked *graph inputs* are rejected at build time (a graph input always has a known rank). A dispatch-time exit gate fails cleanly if any operand is still unranked after inference, so no unranked operand ever reaches a backend. + +- **Folding never reads input data**, as described above, which also bounds what a graph can make the interpreter do. + +- **Bounded constant collection.** The interpreter seeds only from the shape operands of the dynamic operators and walks back along the shape-computation chain, copying only the constants actually on that chain (and skipping oversized constants). Weight tensors are never on a shape chain and are not copied, bounding both memory and folding work (a DoS/OOM guard). + +## Future Consideration + +### Bounded (min/max) dimensions +Currently, the proposed model is intentionally unbounded; a dynamic dimension is either provably static or deferred without a specific size range. As a future enhancement, we plan to allow dynamic dimensions to optionally declare a `minSize` and `maxSize` bound (and potentially an "optimal" size). For example: `{name: 'seqlen', minSize: 1, maxSize: 2048}`. + +Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. + +### Fine-grained Shape Queries +Currently, the `shape()` operator returns an operand's entire shape as a 1D tensor. To provide more fine-grained shape retrieval, we may consider introducing two additional operators: + +- `rank()` → Returns a 0D scalar tensor representing the operand's rank (similar to `tfl.rank`). + +- `dimension(axis)` → Returns a 0D scalar tensor representing the size of a specific dimension (similar to StableHLO's `get_dimension_size`). + +## Open Questions +- **Tensor-Derived sizes.** Whether, and how, to admit dimensions that depend on tensor *values*, which this proposal places out of scope. + +- **How to represent an unnamed dynamic dimension on readback.** On input it is a `null` element of the shape sequence, and the symmetric choice would be `null` on `MLOperand.shape` as well — but a `FrozenArray` cannot hold `null` elements, so the current implementation reports the string `"?"` instead. Whether to keep the `"?"` sentinel, change the readback type to admit `null`, or adopt another representation is left for the Working Group. + +- **Bidirectional broadcasting for Expand.** In dynamic shape scenarios, the shape operand of `expand` may contain flexible dimensions (e.g., `[1, 1, 1]`) that require bidirectional broadcasting against the input tensor. The original WebNN specification was limited to unidirectional broadcasting. Extending the `expand` operator to support bidirectional broadcasting aligns with the ONNX `Expand` operator and ensures correct handling of these dynamic shape cases. + +## References & Acknowledgements +- [Support flexible input sizes #883](https://github.com/webmachinelearning/webnn/issues/883) + +- [ORT API SetDimensions()](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a6575872736b924b47a382deb97e2fc17) + +- [TFLite Flatbuffer Tensor & TfLiteTensor](https://developers.google.com/edge/api/tflite/c/struct/tf-lite-tensor) + +- [Core ML Flexible Input Shapes](https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html) + +- Many thanks for valuable feedback and advice from: + + - [Ningxin Hu](mailto:ningxin.hu@intel.com) + + - [Dwayne Robinson](mailto:dwayner@microsoft.com) From 5424a22c6d780087a5d7399ba6696723f04b898b Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Mon, 24 Aug 2026 16:07:52 +0800 Subject: [PATCH 2/8] Address review feedback on the dynamic shape explainer --- dynamic-shape-explainer.md | 113 +++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 41 deletions(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index b2fcc826..f372c0ca 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -10,13 +10,13 @@ 1. [Introduction](#introduction) 2. [Goals](#goals) 3. [Non-goals](#non-goals) -4. [Use Cases](#non-goals) +4. [Use Cases](#use-cases) 5. [Proposed API](#proposed-api) 6. [Design Discussion](#design-discussion) 7. [Considered Alternatives](#considered-alternatives) 8. [Privacy & Security Considerations](#privacy--security-considerations) 9. [Future Consideration](#future-consideration) -10. [Open Questions](#fine-grained-shape-queries) +10. [Open Questions](#open-questions) 11. [References & Acknowledgements](#references--acknowledgements) ## Introduction @@ -35,7 +35,7 @@ This proposal introduces **Dynamic Shape** for WebNN, enabling graph dimensions ## Goals - Allow a single compiled `MLGraph` to execute across varying runtime input sizes, without rebuilding. -- Model dynamism the way the underlying runtimes already do: a dimension is either a **static size**, a **named dynamic** dimension (a symbolic name), or an **unnamed dynamic** dimension (fully unconstrained). +- A dimension is either a static size or a named dynamic dimension (a symbolic name). Requiring every dynamic dimension to carry a name costs no expressive power and gives every dimension a debuggable symbol. - Defer shape validation that cannot be decided at build time to the point where concrete input shapes are known, without weakening the checks that can be decided at build time. @@ -96,7 +96,7 @@ const padded = builder.padDynamic(x, builder.constant(..., [0, 0]), builder.conc This motivates the **shape-as-data operators** described below. ### Getting output shapes before dispatch -A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs (it is derived inside the subgraph). To allocate the output tensor, the framework needs the concrete output shape before it dispatches: +A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs, it is derived inside the subgraph, and so carries a name the user agent synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: ```js const outShapes = await graph.computeShapes({'attention_mask': [1, 37]}); @@ -106,23 +106,27 @@ const outShapes = await graph.computeShapes({'attention_mask': [1, 37]}); ``` ## Proposed API -The web-facing surface changes fall into three groups: 1) the dimension model and descriptors, 2) new `computeShapes()` methods and 3) new operators. The runtime behavior behind these — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape folding** — is described under [Design Discussion](#design-discussion). +The web-facing surface changes fall into three groups: 1) the dimension model and descriptors, 2) new `computeShapes()` methods and 3) new operators. The runtime behavior behind these — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape computation** — is described under [Design Discussion](#design-discussion). ### 1. Dimension model and descriptors -A graph input's dimension may now be a number (static integer), a string (a **named dynamic** dimension mapped to a runtime symbolic name), or — as a `null` element of the shape sequence — an **unnamed dynamic** dimension. This is expressed with a new `MLDimension` type and a new `MLInputOperandDescriptor`: +A graph input's dimension is either a number (a static integer) or a string (a **named dynamic** dimension mapped to a runtime symbolic name). There is no third state. This is expressed with a new `MLDimension` type and a new `MLInputOperandDescriptor`: ```webidl // A single input dimension: a number is static; a string is a named dynamic -// dimension; a null element (below) is an unnamed dynamic dimension. +// dimension. Every dynamic dimension carries a name. typedef ([EnforceRange] unsigned long or DOMString) MLDimension; dictionary MLInputOperandDescriptor { required MLOperandDataType dataType; - required sequence shape; + required sequence shape; }; ``` -Reading a shape back, `MLOperand.shape` is widened accordingly: its elements may now be strings, an unnamed dynamic dimension surfaces as `"?"` (a provisional representation — see [Open Questions](#fine-grained-shape-queries)), and the whole attribute is `null` for an **unranked** operand — one whose rank is not yet known (see [Dimension semantics](#dimension-semantics)): +The empty string is **not** a valid dimension name; `input()` throws a `TypeError`. A name exists to relate dimensions to one another, and an empty name relates nothing — admitting it would either silently force every `""` dimension in the graph to resolve to the same value, or create a "half-named" dimension whose meaning differs from layer to layer. ONNX uses `dim_param == ""` to mean *anonymous*, so a caller reaching for `""` is asking for the state this design deliberately does not have. + +A framework that receives anonymous dimensions from its own model format synthesizes names before calling WebNN. It already knows a stable identity for each one — the tensor's name and the axis index — so a name such as `"encoder_input_axis1"` is both free to produce and more useful in a diagnostic than an anonymous placeholder. (This is caller-side naming; it is distinct from the names a user agent synthesizes for *derived* dimensions, described in [Dimension semantics](#dimension-semantics).) + +Reading a shape back, `MLOperand.shape` is widened accordingly: its elements may now be strings, and the whole attribute is `null` for an **unranked** operand — one whose rank is not yet known (see [Dimension semantics](#dimension-semantics)): ```webidl interface MLOperand { @@ -132,6 +136,10 @@ interface MLOperand { }; ``` +Because every dynamic dimension is named, an element of this array is always either a number or a non-empty string — there is no sentinel value to interpret. The outer `null` is unrelated: it reports an unranked operand, not an unknown dimension. + +Names that the user agent synthesized for derived dimensions are observable here, which is what makes them useful for debugging, but their content and format are implementation-defined and **not stable**: rebuilding the same graph, or a change in how the implementation walks it, may produce different names. They are diagnostic labels, not identifiers to be read back and fed into `input()` as a cross-tensor constraint. + The plain `MLOperandDescriptor` is deliberately **unchanged** (static-only): it describes `constant()` data, and `computeShapes()` likewise returns fully concrete (static) shapes. Dynamism thus lives only on graph *inputs* and propagates from there. ```webidl @@ -143,7 +151,7 @@ dictionary MLOperandDescriptor { ``` ### 2. `computeShapes()` -`computeShapes()` runs the same shape inference and folding as dispatch, but early and without executing the graph, returning the concrete output shape for each output given concrete input shapes: +`computeShapes()` runs the same shape inference and shape computation as dispatch, but early and without executing the graph, returning the concrete output shape for each output given concrete input shapes: ```webidl partial interface MLGraph { @@ -154,12 +162,32 @@ partial interface MLGraph { The programming model becomes **build → (optionally) computeShapes → dispatch**. -This method enables frameworks to determine the output tensor sizes for dynamic subgraphs whose shapes are difficult to infer. Exposing it reduces redundant computation overhead. +This method enables frameworks to determine the output tensor sizes for dynamic subgraphs whose shapes are difficult to infer. Exposing it reduces redundant computation overhead, and it gives an implementation an opportunity to prepare for the dispatch that follows — see [Shape specialization and preparation](#shape-specialization-and-preparation). ### 3. Shape-as-data operators Threading a dynamic input dimension through the graph is not sufficient on its own; real models compute with shapes. This proposal adds a family of operators that treat a shape as a runtime tensor, plus dynamic variants of existing operators that take their shape parameters as **operands** rather than build-time attributes. ```webidl +dictionary MLSqueezeOptions : MLOperatorOptions { + sequence<[EnforceRange] unsigned long> axes; +}; + +dictionary MLReshapeTo2dOptions : MLOperatorOptions { + [EnforceRange] unsigned long axis = 1; +}; + +dictionary MLSliceDynamicOptions : MLOperatorOptions { + sequence<[EnforceRange] unsigned long> strides; +}; + +// Mirrors MLResample2dOptions, except `sizes` is a runtime operand. +dictionary MLResample2dDynamicOptions : MLOperatorOptions { + MLInterpolationMode mode = "nearest-neighbor"; + sequence scales; + MLOperand sizes; + sequence<[EnforceRange] unsigned long> axes; +}; + partial interface MLGraphBuilder { // Read an operand's shape as a runtime uint32 1-D tensor. MLOperand shape(MLOperand input, optional MLOperatorOptions options = {}); @@ -174,7 +202,7 @@ partial interface MLGraphBuilder { MLOperand squeeze(MLOperand input, optional MLSqueezeOptions options = {}); MLOperand unsqueeze(MLOperand input, sequence<[EnforceRange] unsigned long> axes, optional MLOperatorOptions options = {}); - MLOperand reshapeTo2d(MLOperand input, optional MLReshapeTo2dOptions options = {[EnforceRange] unsigned long axis = 1;}); + MLOperand reshapeTo2d(MLOperand input, optional MLReshapeTo2dOptions options = {}); // Dynamic variants: shape parameters are operands, evaluated at dispatch. MLOperand reshapeDynamic(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {}); @@ -182,17 +210,17 @@ partial interface MLGraphBuilder { MLOperand sliceDynamic(MLOperand input, MLOperand starts, MLOperand sizes, optional MLSliceDynamicOptions options = {}); MLOperand padDynamic(MLOperand input, MLOperand beginningPadding, MLOperand endingPadding, optional MLOperatorOptions options = {}); - // Equal-split (scalar count) and explicit-splits (operand) forms. - sequence splitDynamic(MLOperand input, [EnforceRange] unsigned long splits, optional MLSplitOptions options = {}); sequence splitDynamic(MLOperand input, MLOperand splits, optional MLSplitOptions options = {}); - MLOperand resample2dDynamic(MLOperand input, optional MLResample2dDynamicOptions options = {MLOperand sizes, ...}); + MLOperand resample2dDynamic(MLOperand input, optional MLResample2dDynamicOptions options = {}); MLOperand tileDynamic(MLOperand input, MLOperand repetitions, optional MLOperatorOptions options = {}); }; ``` Three notes on the design of this family, rather than a per-operator walkthrough: -- **Each `*Dynamic` operator is a full dynamic mirror of its static counterpart.** `sliceDynamic` uses the same `starts + sizes (+ strides)` semantics as static `slice`; splitDynamic mirrors split's explicit and equal-split forms and its `axis` option. Keeping them one-to-one lets the static operators stay static-input-only and avoids divergent validation. +- **Each `*Dynamic` operator is a full dynamic mirror of its static counterpart.** `sliceDynamic` uses the same `starts + sizes (+ strides)` semantics as static `slice`; `splitDynamic` mirrors `split`'s explicit-splits form and its `axis` option. Keeping them one-to-one lets the static operators stay static-input-only and avoids divergent validation. + + `splitDynamic` deliberately has **no** equal-split (scalar count) overload: that count is a build-time constant even when the axis is dynamic, so static `split` already expresses it. The divisibility check it would normally perform at build time simply defers to dispatch under the three-valued predicates described below, so no dynamic variant is needed to make it work on a dynamic axis. - **`squeeze` / `unsqueeze` / `reshapeTo2d` are the origin of dynamic rank.** E.g. a no-axes `squeeze` removes every size-1 dimension, so its output rank depends on runtime data. Emulating this logic requires complex subgraph chains to calculate runtime shapes, resulting in significant graph bloat and excessive shape inference overhead. Native operator support provides a more efficient and direct representation of these rank-changing transformations. @@ -210,12 +238,14 @@ For example, when the target shape is a runtime operand (values unknown at build *(Optimization: This chain may be skippable if the framework can prove the operand is already sentinel-free.)* ## Design Discussion -This section describes the runtime behavior that gives it meaning — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape folding**, plus the dimension semantics they honor. +This section describes the runtime behavior that gives it meaning — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape computation**, plus the dimension semantics they honor. The two are easy to confuse by name, so to be precise: *shape inference* propagates each operand's **shape** forward across the whole graph, while *shape computation* evaluates one `shape()`-rooted chain down to the concrete **integer values** a shape parameter needs. ### Dimension semantics -- **Named dimensions are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"). An unnamed dimension carries no such constraint, and a dynamic dimension has **no min/max bound** — anything not provably static simply defers. +- **Shared names are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"); the implementation enforces this across inputs and uses it to cancel dimensions in operations such as `reshape`. A name that appears only once constrains nothing. A dynamic dimension has **no min/max bound** — anything not provably static simply defers. + + Two kinds of names share one namespace: those the **caller** supplies, which may establish identity across tensors, and those the **user agent synthesizes** for derived dimensions, which identify a dimension without implying any relationship. Hence the isolation requirement below. -- **Derived dimensions are never given a synthesized name.** A named dimension survives only on a 1:1 pass-through; any dimension *computed* from a dynamic one becomes unnamed. This avoids inventing symbolic identities the runtime cannot honor. +- **Derived dimensions get a unique synthesized name.** A caller-supplied name survives unchanged only on a 1:1 pass-through; any dimension *computed* from a dynamic one (a `concat` sum, a `conv2d` window, a `resample2d` scale) receives a fresh name that is unique within the graph. Uniqueness is what makes this safe: a name that is guaranteed to appear exactly once asserts no identity with any other dimension, so nothing is claimed that the runtime cannot honor — while the graph still reads back with a symbol at every position, showing where a pass-through ended. - **Unranked operands.** For example, a no-axes `squeeze` removes every size-1 dimension, so its output rank depends on runtime data — the operand is unranked (its `MLOperand.shape` is null) until `computeShapes()`/`dispatch()` recovers it. @@ -241,8 +271,12 @@ bool DimensionsAreDefinitelyUnequal(Dimension a, Dimension b) { The same predicate is applied uniformly across operators that impose cross-dimension constraints, e.g. `matmul`'s contraction dimension, concat's non-concatenated axes, `reshape`'s element-count product, broadcasting, and so on. -### Shape folding at dispatch -*Shape folding* is the dispatch-time evaluation of a `shape()`-rooted chain down to the concrete values a shape parameter needs. It is deliberately narrow — the shape-calculation operations only: arithmetic and structural transforms on shape tensors (usually integer, sometimes float, e.g. reciprocal) — and never the data-producing operators such as `conv2d` or `matmul`, which do not sit on a shape chain. It is the value-computing counterpart to *shape inference* ([Deferred validation](#deferred-validation)), which propagates operand shapes across the whole graph. +### Shape computation at dispatch +*Shape computation* is the dispatch-time evaluation of a `shape()`-rooted chain down to the concrete values a shape parameter needs. It is the value-computing counterpart to *shape inference* ([Deferred validation](#deferred-validation)), which propagates operand shapes across the whole graph. + +There is no blessed list of operators that may appear on a shape chain. In principle any operator can compute a shape — `matmul` reducing two 1-D vectors to an element count is a legitimate, if unusual, way to do it. + +The set the prototype implements — arithmetic and structural transforms on shape tensors, with a little floating-point support for cases such as `reciprocal`, is therefore an implementation-cost and performance trade-off, not a design boundary, and the right balance is something to settle as the feature develops. A small chain makes this concrete. To reshape a `[1, 'seqlen', 512]` tensor into `[1, 'seqlen', 8, 64]` (splitting the static hidden size into 8 heads × 64) while keeping the dynamic sequence length: @@ -254,32 +288,26 @@ const newShape = builder.concat([batchSeq, heads], 0); // [1, seqlen, 8, 64] const y = builder.reshapeDynamic(x, newShape); // y: [1, 'seqlen', 8, 64] ``` -At dispatch with `seqlen = 37`, folding walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That folded value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. +At dispatch with `seqlen = 37`, shape computation walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That computed value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. -What folding may read is bounded by the root: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#fine-grained-shape-queries) case. +What shape computation may read is bounded not by the operators on the chain but by its **root**: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#open-questions) case. ### Backends mapping +The dimension model maps cleanly onto all three backends, as shown below. The **shape-as-data operator family** is currently implemented only on the ORT backend. #### ORT -- **Named dynamic dimension** → ONNX symbolic name ([OrtApi::SetDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a6575872736b924b47a382deb97e2fc17)): +- **Dynamic dimension** → ONNX symbolic name ([OrtApi::SetSymbolicDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#aa5b1654064d833a515f3acfcdcc5e81d)): ```cpp WebNN: shape=['batch', 512] ONNX: shape=[dim_param:'batch', 512] ``` -- **Unnamed dynamic dimension** → ONNX free dimension ([OrtApi::SetSymbolicDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#aa5b1654064d833a515f3acfcdcc5e81d)): - -```cpp -WebNN: shape=[null, 512] -ONNX: shape=[-1, 512] -``` - #### LiteRT -- **Named/Unnamed dynamic dimension** → LiteRT unknown dimensions: +- **Dynamic dimension** → LiteRT unknown dimensions: ```cpp -WebNN: shape=["batch", null, 512] +WebNN: shape=["batch", "height", 512] LiteRT: shape=[-1, -1, 512] ``` @@ -290,11 +318,11 @@ const flatbuffers::Offset> dimensions ``` #### Core ML -- **Named/Unnamed dynamic dimension** → Core ML unbounded ranges dimensions: +- **Dynamic dimension** → Core ML unbounded ranges dimensions: ```cpp WebNN: - shape=["batch", null, 512] + shape=["batch", "height", 512] ``` ```cpp @@ -314,22 +342,27 @@ The first cut loosened each operator's build-time validator independently to tol We considered giving each `*Dynamic` operator a shape signature tailored to its most common use, rather than mirroring the static operator. We rejected it for the reasons in [Shape-as-data operators](#3-shape-as-data-operators): faithful mirroring keeps the mental model small and lets the static operators stay static-input-only. ## Privacy & Security Considerations -Dynamic shapes add no new fingerprinting or cross-origin surface: they expose no device or environment information a static graph does not, and folding and inference read only shapes the page itself supplied. The considerations below are therefore about security. +Dynamic shapes add no new fingerprinting or cross-origin surface: they expose no device or environment information a static graph does not, and shape computation and inference read only shapes the page itself supplied. The synthesized names for derived dimensions are newly observable output, but they are derived from the graph the page itself constructed (operation type, index, and axis) and carry nothing about the device or the environment; they are also explicitly unstable and not part of the API contract, so nothing may be inferred from a change in one. The considerations below are therefore about security. The renderer is untrusted, so the service must remain safe on any graph a compromised renderer can construct, including ill-formed dynamic graphs: - **No dereference of an absent rank.** Unranked operands (e.g. from a no-axes squeeze) are handled uniformly by each shared validator — propagate, resolve, or cleanly reject — and unranked *graph inputs* are rejected at build time (a graph input always has a known rank). A dispatch-time exit gate fails cleanly if any operand is still unranked after inference, so no unranked operand ever reaches a backend. -- **Folding never reads input data**, as described above, which also bounds what a graph can make the interpreter do. +- **Shape computation never reads input data**, as described above, which also bounds what a graph can make the interpreter do. -- **Bounded constant collection.** The interpreter seeds only from the shape operands of the dynamic operators and walks back along the shape-computation chain, copying only the constants actually on that chain (and skipping oversized constants). Weight tensors are never on a shape chain and are not copied, bounding both memory and folding work (a DoS/OOM guard). +- **Bounded constant collection.** The interpreter seeds only from the shape operands of the dynamic operators and walks back along the shape-computation chain, copying only the constants actually on that chain (and skipping oversized constants). Weight tensors are never on a shape chain and are not copied, bounding both memory and shape-computation work (a DoS/OOM guard). ## Future Consideration ### Bounded (min/max) dimensions Currently, the proposed model is intentionally unbounded; a dynamic dimension is either provably static or deferred without a specific size range. As a future enhancement, we plan to allow dynamic dimensions to optionally declare a `minSize` and `maxSize` bound (and potentially an "optimal" size). For example: `{name: 'seqlen', minSize: 1, maxSize: 2048}`. -Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. +Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. Some runtime already accepts a constraint of this shape and acts on it. TensorRT's optimization profiles (min / opt / max), Core ML's `RangeDim`, and OpenVINO's bounded partial shapes, so the hint has somewhere to go rather than terminating in the user agent. + +### Shape specialization and preparation +Knowing the shapes is not the same as being ready to run them: a backend may still have to plan memory, select kernels, or recompile. What that costs varies a great deal between runtimes (some absorb a shape change almost for free), while others may re-compile the graph. So on some backends a caller that changes shape often pays a real price. + +`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named the shapes it is about to run. That helps when a shape is then reused, but not when a caller keeps switching between shapes, and a user agent cannot tell which of them are worth keeping ready. One direction worth exploring is to let the caller say so: build with difference input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. ### Fine-grained Shape Queries Currently, the `shape()` operator returns an operand's entire shape as a 1D tensor. To provide more fine-grained shape retrieval, we may consider introducing two additional operators: @@ -341,8 +374,6 @@ Currently, the `shape()` operator returns an operand's entire shape as a 1D tens ## Open Questions - **Tensor-Derived sizes.** Whether, and how, to admit dimensions that depend on tensor *values*, which this proposal places out of scope. -- **How to represent an unnamed dynamic dimension on readback.** On input it is a `null` element of the shape sequence, and the symmetric choice would be `null` on `MLOperand.shape` as well — but a `FrozenArray` cannot hold `null` elements, so the current implementation reports the string `"?"` instead. Whether to keep the `"?"` sentinel, change the readback type to admit `null`, or adopt another representation is left for the Working Group. - - **Bidirectional broadcasting for Expand.** In dynamic shape scenarios, the shape operand of `expand` may contain flexible dimensions (e.g., `[1, 1, 1]`) that require bidirectional broadcasting against the input tensor. The original WebNN specification was limited to unidirectional broadcasting. Extending the `expand` operator to support bidirectional broadcasting aligns with the ONNX `Expand` operator and ensures correct handling of these dynamic shape cases. ## References & Acknowledgements From 8267b1c629f908c1e16b3d7f067f2785e58b377d Mon Sep 17 00:00:00 2001 From: Dwayne Robinson Date: Thu, 27 Aug 2026 16:40:01 -0700 Subject: [PATCH 3/8] Update dynamic-shape-explainer.md with minor grammar --- dynamic-shape-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index f372c0ca..11c55927 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -96,7 +96,7 @@ const padded = builder.padDynamic(x, builder.constant(..., [0, 0]), builder.conc This motivates the **shape-as-data operators** described below. ### Getting output shapes before dispatch -A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs, it is derived inside the subgraph, and so carries a name the user agent synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: +A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs - it is derived inside the subgraph, and so carries a name the user agent synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: ```js const outShapes = await graph.computeShapes({'attention_mask': [1, 37]}); From 4501ef431959e46795c68289b95c61c742aee37e Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Fri, 28 Aug 2026 23:04:04 +0800 Subject: [PATCH 4/8] Address second review round on the dynamic shape explainer --- dynamic-shape-explainer.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index 11c55927..6364fbfe 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -13,6 +13,10 @@ 4. [Use Cases](#use-cases) 5. [Proposed API](#proposed-api) 6. [Design Discussion](#design-discussion) + - [Dimension semantics](#dimension-semantics) + - [Deferred validation](#deferred-validation) + - [Shape computation at dispatch](#shape-computation-at-dispatch) + - [Backends mapping](#backends-mapping) 7. [Considered Alternatives](#considered-alternatives) 8. [Privacy & Security Considerations](#privacy--security-considerations) 9. [Future Consideration](#future-consideration) @@ -96,10 +100,10 @@ const padded = builder.padDynamic(x, builder.constant(..., [0, 0]), builder.conc This motivates the **shape-as-data operators** described below. ### Getting output shapes before dispatch -A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs - it is derived inside the subgraph, and so carries a name the user agent synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: +A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs — it is derived inside the subgraph, and so carries a name the user agent synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: ```js -const outShapes = await graph.computeShapes({'attention_mask': [1, 37]}); +const outShapes = graph.computeShapes({'attention_mask': [1, 37]}); // => {'output': [1, 74]} // Allocate output MLTensors of the resolved size, then dispatch. @@ -238,7 +242,7 @@ For example, when the target shape is a runtime operand (values unknown at build *(Optimization: This chain may be skippable if the framework can prove the operand is already sentinel-free.)* ## Design Discussion -This section describes the runtime behavior that gives it meaning — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape computation**, plus the dimension semantics they honor. The two are easy to confuse by name, so to be precise: *shape inference* propagates each operand's **shape** forward across the whole graph, while *shape computation* evaluates one `shape()`-rooted chain down to the concrete **integer values** a shape parameter needs. +This section describes the runtime behavior that gives it meaning, plus the dimension semantics it honors. Two mechanisms carry dynamism at dispatch: **shape inference**, which walks the whole graph and computes each operand's *shape* ([Deferred validation](#deferred-validation)), and **shape computation**, which evaluates a single `shape()`-rooted chain to the *values* a shape parameter needs ([Shape computation at dispatch](#shape-computation-at-dispatch)). The second is a step inside the first. ### Dimension semantics - **Shared names are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"); the implementation enforces this across inputs and uses it to cancel dimensions in operations such as `reshape`. A name that appears only once constrains nothing. A dynamic dimension has **no min/max bound** — anything not provably static simply defers. @@ -252,9 +256,9 @@ This section describes the runtime behavior that gives it meaning — the two me ### Deferred validation Validation splits across two phases: -- **At build time**, we validate only what is knowable without concrete shapes: data-type compatibility, rank constraints (e.g. `conv2d` needs rank 4), same-name symbolic consistency, and any **definite static contradiction** (e.g. reshaping a static `[2, 3]` to `[7]`). +- **At build time**, shapes propagate symbolically — every operand gets a shape expressed in names and static sizes — and we validate only what is knowable without concrete shapes: data-type compatibility, rank constraints (e.g. `conv2d` needs rank 4), same-name symbolic consistency, and any **definite static contradiction** (e.g. reshaping a static `[2, 3]` to `[7]`). -- **At dispatch time** (and at `computeShapes()`), once concrete input shapes are known, we run **shape inference** — a forward propagation of each operand's concrete *shape* — over the whole graph, and validate the resulting concrete shapes against every constraint, including buffer sizes. +- **At dispatch time** (and at `computeShapes()`), once concrete input shapes are known, we run **shape inference**: the same forward propagation, now carrying each operand's concrete *shape* across the whole graph. We then validate the resulting concrete shapes against every constraint, including buffer sizes. Deferring this work is the inherent trade-off of dynamic shapes: the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so a user agent should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. @@ -272,7 +276,7 @@ bool DimensionsAreDefinitelyUnequal(Dimension a, Dimension b) { The same predicate is applied uniformly across operators that impose cross-dimension constraints, e.g. `matmul`'s contraction dimension, concat's non-concatenated axes, `reshape`'s element-count product, broadcasting, and so on. ### Shape computation at dispatch -*Shape computation* is the dispatch-time evaluation of a `shape()`-rooted chain down to the concrete values a shape parameter needs. It is the value-computing counterpart to *shape inference* ([Deferred validation](#deferred-validation)), which propagates operand shapes across the whole graph. +*Shape computation* is performed by a **shape interpreter** in the user agent, which evaluates a `shape()`-rooted chain down to the concrete integer values a shape parameter needs. Shape inference ([Deferred validation](#deferred-validation)) invokes it at dispatch each time it reaches a `*Dynamic` operator, and uses the values it returns as that operator's output shape. There is no blessed list of operators that may appear on a shape chain. In principle any operator can compute a shape — `matmul` reducing two 1-D vectors to an element count is a legitimate, if unusual, way to do it. @@ -288,9 +292,9 @@ const newShape = builder.concat([batchSeq, heads], 0); // [1, seqlen, 8, 64] const y = builder.reshapeDynamic(x, newShape); // y: [1, 'seqlen', 8, 64] ``` -At dispatch with `seqlen = 37`, shape computation walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That computed value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. +At dispatch with `seqlen = 37`, the interpreter walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That computed value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. -What shape computation may read is bounded not by the operators on the chain but by its **root**: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#open-questions) case. +What the interpreter may read is bounded not by the operators on the chain but by its **root**: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#open-questions) case. ### Backends mapping The dimension model maps cleanly onto all three backends, as shown below. The **shape-as-data operator family** is currently implemented only on the ORT backend. @@ -346,23 +350,23 @@ Dynamic shapes add no new fingerprinting or cross-origin surface: they expose no The renderer is untrusted, so the service must remain safe on any graph a compromised renderer can construct, including ill-formed dynamic graphs: -- **No dereference of an absent rank.** Unranked operands (e.g. from a no-axes squeeze) are handled uniformly by each shared validator — propagate, resolve, or cleanly reject — and unranked *graph inputs* are rejected at build time (a graph input always has a known rank). A dispatch-time exit gate fails cleanly if any operand is still unranked after inference, so no unranked operand ever reaches a backend. +- **No dereference of an absent rank.** Unranked operands (e.g. from a no-axes squeeze) are handled uniformly by each shared validator — propagate, resolve, or cleanly reject — and unranked *graph inputs* are rejected at build time (a graph input always has a known rank). A dispatch-time exit gate fails cleanly if any operand is still unranked after shape inference, so no unranked operand ever reaches a backend. - **Shape computation never reads input data**, as described above, which also bounds what a graph can make the interpreter do. -- **Bounded constant collection.** The interpreter seeds only from the shape operands of the dynamic operators and walks back along the shape-computation chain, copying only the constants actually on that chain (and skipping oversized constants). Weight tensors are never on a shape chain and are not copied, bounding both memory and shape-computation work (a DoS/OOM guard). +- **Bounded constant collection.** The constants that the interpreter may need are gathered ahead of time: the collector seeds from the shape operands of the dynamic operators, walks back along the chain, and copies only the constants it finds there. Current weight tensors are not on a shape chain and are not copied, which keeps the memory and work involved independent of the model's size (a DoS/OOM guard). ## Future Consideration ### Bounded (min/max) dimensions Currently, the proposed model is intentionally unbounded; a dynamic dimension is either provably static or deferred without a specific size range. As a future enhancement, we plan to allow dynamic dimensions to optionally declare a `minSize` and `maxSize` bound (and potentially an "optimal" size). For example: `{name: 'seqlen', minSize: 1, maxSize: 2048}`. -Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. Some runtime already accepts a constraint of this shape and acts on it. TensorRT's optimization profiles (min / opt / max), Core ML's `RangeDim`, and OpenVINO's bounded partial shapes, so the hint has somewhere to go rather than terminating in the user agent. +Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. Several runtimes already accept a constraint of this shape and act on it — TensorRT's optimization profiles (min / opt / max), Core ML's `RangeDim`, and OpenVINO's bounded partial shapes — so the hint has somewhere to go rather than terminating in the user agent. ### Shape specialization and preparation -Knowing the shapes is not the same as being ready to run them: a backend may still have to plan memory, select kernels, or recompile. What that costs varies a great deal between runtimes (some absorb a shape change almost for free), while others may re-compile the graph. So on some backends a caller that changes shape often pays a real price. +Knowing the shapes is not the same as being ready to run them: a backend may still have to plan memory, select kernels, or recompile. What that costs varies a great deal between runtimes: some absorb a shape change almost for free, while others re-compile the graph. So on some backends a caller that changes shape often pays a real price. -`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named the shapes it is about to run. That helps when a shape is then reused, but not when a caller keeps switching between shapes, and a user agent cannot tell which of them are worth keeping ready. One direction worth exploring is to let the caller say so: build with difference input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. +`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named the shapes it is about to run. That helps when a shape is then reused, but not when a caller keeps switching between shapes, and a user agent cannot tell which of them are worth keeping ready. One direction worth exploring is to let the caller say so: build with different input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. ### Fine-grained Shape Queries Currently, the `shape()` operator returns an operand's entire shape as a 1D tensor. To provide more fine-grained shape retrieval, we may consider introducing two additional operators: From 900b1e896980a82cda5067493bdf39ffa208a4b6 Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Mon, 31 Aug 2026 23:00:52 +0800 Subject: [PATCH 5/8] Address more comments --- dynamic-shape-explainer.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index 6364fbfe..725a58a0 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -242,7 +242,11 @@ For example, when the target shape is a runtime operand (values unknown at build *(Optimization: This chain may be skippable if the framework can prove the operand is already sentinel-free.)* ## Design Discussion -This section describes the runtime behavior that gives it meaning, plus the dimension semantics it honors. Two mechanisms carry dynamism at dispatch: **shape inference**, which walks the whole graph and computes each operand's *shape* ([Deferred validation](#deferred-validation)), and **shape computation**, which evaluates a single `shape()`-rooted chain to the *values* a shape parameter needs ([Shape computation at dispatch](#shape-computation-at-dispatch)). The second is a step inside the first. +This section describes the runtime behavior that gives it meaning, plus the dimension semantics it honors. Two mechanisms carry dynamism at dispatch: + +`Shape Inference` is the whole-graph pass: it computes each operand's shape, over symbolic dimensions at build time and over concrete ones at dispatch. + +`Shape Computation` isn't a superset of it — it's a step inside it, and it computes a different kind of thing. When inference reaches reshapeDynamic(x, newShape), newShape's own shape (a 1-D tensor of length 4, say) tells it nothing about the output; it needs newShape's values. So inference pauses, evaluates the chain that produces newShape down to concrete integers, and resumes with that as the output shape. Whole graph vs. one chain and inference is the caller. ### Dimension semantics - **Shared names are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"); the implementation enforces this across inputs and uses it to cancel dimensions in operations such as `reshape`. A name that appears only once constrains nothing. A dynamic dimension has **no min/max bound** — anything not provably static simply defers. From 44ceb9f005978f6fad49837b01e8a26bbea05883 Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Wed, 9 Sep 2026 17:59:06 +0800 Subject: [PATCH 6/8] Address review comments on deferred validation, computeShapes and fingerprinting --- dynamic-shape-explainer.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index 725a58a0..1b311f2c 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -155,7 +155,7 @@ dictionary MLOperandDescriptor { ``` ### 2. `computeShapes()` -`computeShapes()` runs the same shape inference and shape computation as dispatch, but early and without executing the graph, returning the concrete output shape for each output given concrete input shapes: +`computeShapes()` runs the same shape inference and shape computation as dispatch, but early and without executing the graph: one forward pass over the graph resolves every operand, and it returns the concrete shape of each output for the given input shapes: ```webidl partial interface MLGraph { @@ -164,9 +164,9 @@ partial interface MLGraph { }; ``` -The programming model becomes **build → (optionally) computeShapes → dispatch**. +The programming model becomes **build → (optionally) computeShapes → dispatch**. `dispatch()` resolves shapes itself regardless: `computeShapes()` is optional, and the tensors actually bound at dispatch may differ from the shapes it was asked about. Because the result is a pure function of the input shapes, an implementation can cache it, so a repeat is a lookup rather than a second pass. -This method enables frameworks to determine the output tensor sizes for dynamic subgraphs whose shapes are difficult to infer. Exposing it reduces redundant computation overhead, and it gives an implementation an opportunity to prepare for the dispatch that follows — see [Shape specialization and preparation](#shape-specialization-and-preparation). +Frameworks need this because `dispatch()` takes caller-allocated output tensors: the output shape has to be known *before* the call, not during it. Without `computeShapes()`, a framework would have to reimplement WebNN's shape inference just to size those allocations — and for a subgraph whose dynamic dimensions are derived internally, that means reproducing the whole shape chain. The method exposes the resolution that `dispatch()` performs anyway, early enough to be useful, and it gives an implementation an opportunity to prepare for a dispatch that may follow — see [Shape specialization and preparation](#shape-specialization-and-preparation). ### 3. Shape-as-data operators Threading a dynamic input dimension through the graph is not sufficient on its own; real models compute with shapes. This proposal adds a family of operators that treat a shape as a runtime tensor, plus dynamic variants of existing operators that take their shape parameters as **operands** rather than build-time attributes. @@ -264,7 +264,7 @@ Validation splits across two phases: - **At dispatch time** (and at `computeShapes()`), once concrete input shapes are known, we run **shape inference**: the same forward propagation, now carrying each operand's concrete *shape* across the whole graph. We then validate the resulting concrete shapes against every constraint, including buffer sizes. -Deferring this work is the inherent trade-off of dynamic shapes: the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so a user agent should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. +Some deferral is inherent: a constraint whose truth depends on a concrete value cannot be settled before that value exists. How *much* is deferred is a design choice, and this proposal defers nearly all of it — a dynamic dimension is an opaque name rather than a symbolic expression, so build time can only reject contradictions provable between names ([Symbolic shape expressions](#symbolic-shape-expressions) covers the alternative). So the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so a user agent should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. The build-time checks are expressed as **three-valued** dimension predicates. Instead of "equal / not-equal", a comparison is *provably-equal*, *provably-unequal*, or *unknown (defer)*. Only a provable contradiction is rejected at build time: @@ -349,8 +349,21 @@ The first cut loosened each operator's build-time validator independently to tol ### Bespoke dynamic operators We considered giving each `*Dynamic` operator a shape signature tailored to its most common use, rather than mirroring the static operator. We rejected it for the reasons in [Shape-as-data operators](#3-shape-as-data-operators): faithful mirroring keeps the mental model small and lets the static operators stay static-input-only. +### Symbolic shape expressions +A stronger alternative is to carry a symbolic *expression* for every dimension rather than an opaque name, so that build time can prove or refute a graph's shape constraints, report the constraints a model actually requires (divisibility, bounds), and reduce `computeShapes()` to substitution. TensorRT and NNEF's SkriptND both work this way. + +We start with opaque names for three reasons. Shape-as-data operators put some shapes beyond any closed-form expression: a target shape is a runtime operand produced by an arbitrary chain, including data-dependent selection such as the `where` used to lower ONNX's `-1`, and a no-axes `squeeze` leaves even the rank data-dependent — so symbolic inference could cover a large subset of graphs but could not replace the dispatch-time gate. Backends cannot consume expressions either: ONNX takes a `dim_param` string, LiteRT a `-1`, Core ML a `RangeDim`, so expressions would remain a user-agent-internal enrichment. And because the graph comes from an untrusted renderer, build-time proving is attacker-influenced work in a privileged process; bounding it by wall-clock time would make build outcomes machine-dependent, so a deterministic budget on expression size and depth would be needed instead. + +Expressions stay attractive as a later, additive layer — for diagnostics and for earlier rejection — and pair naturally with [Bounded (min/max) dimensions](#bounded-minmax-dimensions), which supply the input constraints most build-time proofs need. + ## Privacy & Security Considerations -Dynamic shapes add no new fingerprinting or cross-origin surface: they expose no device or environment information a static graph does not, and shape computation and inference read only shapes the page itself supplied. The synthesized names for derived dimensions are newly observable output, but they are derived from the graph the page itself constructed (operation type, index, and axis) and carry nothing about the device or the environment; they are also explicitly unstable and not part of the API contract, so nothing may be inferred from a change in one. The considerations below are therefore about security. +Dynamic shapes add no new **declarative** fingerprinting or cross-origin surface: no value the API returns carries device or environment information a static graph does not, and shape computation and inference read only shapes the page itself supplied. The synthesized names for derived dimensions are newly observable output, but they are derived from the graph the page itself constructed (operation type, index, and axis) and carry nothing about the device or the environment; they are also explicitly unstable and not part of the API contract, so nothing may be inferred from a change in one. + +They do widen an existing **timing** surface. Because one graph now serves many shapes, a page can dispatch it over a swept range of shapes and time each one, observing a cost *curve* where a static graph exposes only the single duration of its `build()`. Discontinuities in that curve — an alignment or tiling threshold, or a shape that falls back to a slower path — can suggest the backend in use or the class of hardware behind it. This is the same class of signal as the compilation timing already observable from `build()`, and from shader compilation in other web APIs, though on a device where WebNN reaches an accelerator no other web API exposes it is a residual signal rather than a duplicate of one. It cannot be fully mitigated in an API whose purpose is to run computation on device-specific accelerators: constant-time execution across shapes is not a realistic option. + +Two things bound it. The work WebNN itself adds at dispatch — shape inference and validation — is data-independent, and its result is cached per set of input shapes, so a repeated shape costs the same; the variable part is the backend's own re-specialization, which the underlying runtime performs with or without WebNN. And explicit specialization ([Shape specialization and preparation](#shape-specialization-and-preparation)) would attribute that cost to a step the caller asked for, rather than leaving it implicit at dispatch. + +The considerations below are therefore about security. The renderer is untrusted, so the service must remain safe on any graph a compromised renderer can construct, including ill-formed dynamic graphs: @@ -370,7 +383,7 @@ Rather than introducing a new source of dynamism, these bounds will serve as cri ### Shape specialization and preparation Knowing the shapes is not the same as being ready to run them: a backend may still have to plan memory, select kernels, or recompile. What that costs varies a great deal between runtimes: some absorb a shape change almost for free, while others re-compile the graph. So on some backends a caller that changes shape often pays a real price. -`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named the shapes it is about to run. That helps when a shape is then reused, but not when a caller keeps switching between shapes, and a user agent cannot tell which of them are worth keeping ready. One direction worth exploring is to let the caller say so: build with different input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. +`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named a set of concrete shapes. How good a predictor that is depends on why it was called: a framework sizing output tensors for a dispatch it is about to make is one thing, a caller comparing candidate output shapes and running none of them is another, and an implementation cannot distinguish them. The call is also synchronous, which bounds what it can reasonably do there. Even where preparation is warranted it pays off only if the shape is reused, and a user agent has no way to know which shapes are worth keeping ready. One direction worth exploring is to let the caller say so: build with different input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. ### Fine-grained Shape Queries Currently, the `shape()` operator returns an operand's entire shape as a 1D tensor. To provide more fine-grained shape retrieval, we may consider introducing two additional operators: From 045d2a786ebfaa810a87eec8e6ef8d81c1ddc120 Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Thu, 10 Sep 2026 18:53:36 +0800 Subject: [PATCH 7/8] Clarify shape computation requirements and record open questions --- dynamic-shape-explainer.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index 1b311f2c..b14c6ebd 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -158,9 +158,10 @@ dictionary MLOperandDescriptor { `computeShapes()` runs the same shape inference and shape computation as dispatch, but early and without executing the graph: one forward pass over the graph resolves every operand, and it returns the concrete shape of each output for the given input shapes: ```webidl +typedef record> MLNamedShapes; + partial interface MLGraph { - record> computeShapes( - record> inputShapes); + MLNamedShapes computeShapes(MLNamedShapes inputShapes); }; ``` @@ -279,12 +280,18 @@ bool DimensionsAreDefinitelyUnequal(Dimension a, Dimension b) { The same predicate is applied uniformly across operators that impose cross-dimension constraints, e.g. `matmul`'s contraction dimension, concat's non-concatenated axes, `reshape`'s element-count product, broadcasting, and so on. +This invariant is one-sided, and only its upper bound is a contract: an implementation **must not** reject a graph that some assignment of concrete input shapes would make valid. Nothing requires it to be good at detecting the rest. The predicate above is the weakest way to honor that — it proves a contradiction only between two static dimensions — and an implementation carrying symbolic *expressions* may reject more, and earlier. Concatenating two `[batch, seq]` tensors along axis 1 doubles the element count, for instance, so reshaping that result back to `[batch, seq]` cannot hold for any input: our model sees only an opaque name for the concatenated axis and defers, while an implementation that knows the axis is `2 * seq` can prove the graph dead and may fail `build()` on it — as some backends performing this analysis internally already do. That is legitimate and costs nothing in portability, since such a graph would fail at every dispatch in any case. It does mean that a successful `build()` is not a portable promise that a graph will ever run — only that this implementation could not prove otherwise. + ### Shape computation at dispatch *Shape computation* is performed by a **shape interpreter** in the user agent, which evaluates a `shape()`-rooted chain down to the concrete integer values a shape parameter needs. Shape inference ([Deferred validation](#deferred-validation)) invokes it at dispatch each time it reaches a `*Dynamic` operator, and uses the values it returns as that operator's output shape. -There is no blessed list of operators that may appear on a shape chain. In principle any operator can compute a shape — `matmul` reducing two 1-D vectors to an element count is a legitimate, if unusual, way to do it. +Two questions hide behind "which operators may appear on a shape chain": what a graph may *express*, and what an implementation must be able to *resolve*. + +The first is not restricted by operator identity. In principle any operator can compute a shape — `matmul` reducing two 1-D vectors to an element count is a legitimate, if unusual, way to do it. What is restricted is the chain's **root**, described below, and that restriction is what keeps a shape chain from becoming the model. -The set the prototype implements — arithmetic and structural transforms on shape tensors, with a little floating-point support for cases such as `reciprocal`, is therefore an implementation-cost and performance trade-off, not a design boundary, and the right balance is something to settle as the feature develops. +Interoperability lives in the second question, and it needs a floor: shapes that resolve in one user agent should resolve in another. ONNX answers the same question with a deliberately narrow closed set — the operators carrying a data-propagation function (`Add`, `Cast`, `Concat`, `Gather`, `Mul`, `Shape`, `Size`, `Slice`, `Squeeze`, `Sub`, `Unsqueeze`) — while ORT's [`symbolic_shape_infer.py`](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py) covers a much wider de-facto set. A required minimum for WebNN belongs between the two, and naming it is work this proposal still owes; the prototype today implements a strict superset of the ONNX set — arithmetic and structural transforms on shape vectors, plus enough floating-point for cases such as `reciprocal`. + +Above that floor, a stronger interpreter is a quality-of-implementation matter, with one requirement: an implementation that cannot evaluate a chain must fail with a clean "cannot resolve this shape" error, rather than executing the chain on the accelerator or guessing a value. Heavy operators — `conv2d`, `matmul` on real tensors, attention — are not expected to be resolvable, and an implementation may reject them on a shape chain. Evaluation is integer work on small vectors in the user agent; it is not an execution of the graph, and it never reads a tensor's data. A small chain makes this concrete. To reshape a `[1, 'seqlen', 512]` tensor into `[1, 'seqlen', 8, 64]` (splitting the static hidden size into 8 heads × 64) while keeping the dynamic sequence length: @@ -393,7 +400,9 @@ Currently, the `shape()` operator returns an operand's entire shape as a 1D tens - `dimension(axis)` → Returns a 0D scalar tensor representing the size of a specific dimension (similar to StableHLO's `get_dimension_size`). ## Open Questions -- **Tensor-Derived sizes.** Whether, and how, to admit dimensions that depend on tensor *values*, which this proposal places out of scope. +- **Tensor-Derived sizes.** Whether, and how, to admit dimensions that depend on tensor *values* — an output whose extent is `NonZero`-shaped rather than a function of the input shapes. This proposal places them out of scope, and draws that boundary structurally rather than by classifying tensors: follow a shape chain back and it bottoms out either at `shape()` outputs and build-time constants, in which case it resolves before dispatch, or at a tensor's data, in which case it is rejected. Admitting the second class needs more than interpreter coverage. Something has to allocate the output before its size is known, which requires an upper bound to allocate against (see [Bounded (min/max) dimensions](#bounded-minmax-dimensions)) and a way to report the size actually produced; and `computeShapes()` could not answer for such an output at all, since the answer does not exist until the graph runs. + +- **Making the shape subgraph explicit.** The values on a shape chain are already separate from tensor data by construction: a chain's roots are `shape()` outputs and build-time constants, so nothing on it can read a tensor, and an implementation resolves it on CPU without executing the graph. That separation is *implicit* today — it follows from the root restriction rather than from anything in the type system, and `MLOperand` is reused for both — which has already led readers to expect that `computeShapes()` might have to run the model. Options for making it visible: expose an operand property such as `isShape`, which documents the split and adds no constraints; or introduce a distinct `MLShapeOperand` type, making the guarantee structural. - **Bidirectional broadcasting for Expand.** In dynamic shape scenarios, the shape operand of `expand` may contain flexible dimensions (e.g., `[1, 1, 1]`) that require bidirectional broadcasting against the input tensor. The original WebNN specification was limited to unidirectional broadcasting. Extending the `expand` operator to support bidirectional broadcasting aligns with the ONNX `Expand` operator and ensures correct handling of these dynamic shape cases. From c2140e3d63f83da638552dfc46db9037ed6cbc3f Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Mon, 14 Sep 2026 17:46:20 +0800 Subject: [PATCH 8/8] State the shape subgraph boundary explicitly and simplify wording --- dynamic-shape-explainer.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md index b14c6ebd..20c8c58f 100644 --- a/dynamic-shape-explainer.md +++ b/dynamic-shape-explainer.md @@ -100,7 +100,7 @@ const padded = builder.padDynamic(x, builder.constant(..., [0, 0]), builder.conc This motivates the **shape-as-data operators** described below. ### Getting output shapes before dispatch -A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs — it is derived inside the subgraph, and so carries a name the user agent synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: +A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs — it is derived inside the subgraph, and so carries a name the implementation synthesized rather than one the framework supplied. To allocate the output tensor, the framework needs the concrete output shape before it dispatches: ```js const outShapes = graph.computeShapes({'attention_mask': [1, 37]}); @@ -128,7 +128,7 @@ dictionary MLInputOperandDescriptor { The empty string is **not** a valid dimension name; `input()` throws a `TypeError`. A name exists to relate dimensions to one another, and an empty name relates nothing — admitting it would either silently force every `""` dimension in the graph to resolve to the same value, or create a "half-named" dimension whose meaning differs from layer to layer. ONNX uses `dim_param == ""` to mean *anonymous*, so a caller reaching for `""` is asking for the state this design deliberately does not have. -A framework that receives anonymous dimensions from its own model format synthesizes names before calling WebNN. It already knows a stable identity for each one — the tensor's name and the axis index — so a name such as `"encoder_input_axis1"` is both free to produce and more useful in a diagnostic than an anonymous placeholder. (This is caller-side naming; it is distinct from the names a user agent synthesizes for *derived* dimensions, described in [Dimension semantics](#dimension-semantics).) +A framework that receives anonymous dimensions from its own model format synthesizes names before calling WebNN. It already knows a stable identity for each one — the tensor's name and the axis index — so a name such as `"encoder_input_axis1"` is both free to produce and more useful in a diagnostic than an anonymous placeholder. (This is caller-side naming; it is distinct from the names an implementation synthesizes for *derived* dimensions, described in [Dimension semantics](#dimension-semantics).) Reading a shape back, `MLOperand.shape` is widened accordingly: its elements may now be strings, and the whole attribute is `null` for an **unranked** operand — one whose rank is not yet known (see [Dimension semantics](#dimension-semantics)): @@ -142,7 +142,7 @@ interface MLOperand { Because every dynamic dimension is named, an element of this array is always either a number or a non-empty string — there is no sentinel value to interpret. The outer `null` is unrelated: it reports an unranked operand, not an unknown dimension. -Names that the user agent synthesized for derived dimensions are observable here, which is what makes them useful for debugging, but their content and format are implementation-defined and **not stable**: rebuilding the same graph, or a change in how the implementation walks it, may produce different names. They are diagnostic labels, not identifiers to be read back and fed into `input()` as a cross-tensor constraint. +Names that the implementation synthesized for derived dimensions are observable here, which is what makes them useful for debugging, but their content and format are implementation-defined and **not stable**: rebuilding the same graph, or a change in how the implementation walks it, may produce different names. They are diagnostic labels, not identifiers to be read back and fed into `input()` as a cross-tensor constraint. The plain `MLOperandDescriptor` is deliberately **unchanged** (static-only): it describes `constant()` data, and `computeShapes()` likewise returns fully concrete (static) shapes. Dynamism thus lives only on graph *inputs* and propagates from there. @@ -252,7 +252,7 @@ This section describes the runtime behavior that gives it meaning, plus the dime ### Dimension semantics - **Shared names are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"); the implementation enforces this across inputs and uses it to cancel dimensions in operations such as `reshape`. A name that appears only once constrains nothing. A dynamic dimension has **no min/max bound** — anything not provably static simply defers. - Two kinds of names share one namespace: those the **caller** supplies, which may establish identity across tensors, and those the **user agent synthesizes** for derived dimensions, which identify a dimension without implying any relationship. Hence the isolation requirement below. + Two kinds of names share one namespace: those the **caller** supplies, which may establish identity across tensors, and those the **implementation synthesizes** for derived dimensions, which identify a dimension without implying any relationship. Hence the isolation requirement below. - **Derived dimensions get a unique synthesized name.** A caller-supplied name survives unchanged only on a 1:1 pass-through; any dimension *computed* from a dynamic one (a `concat` sum, a `conv2d` window, a `resample2d` scale) receives a fresh name that is unique within the graph. Uniqueness is what makes this safe: a name that is guaranteed to appear exactly once asserts no identity with any other dimension, so nothing is claimed that the runtime cannot honor — while the graph still reads back with a symbol at every position, showing where a pass-through ended. @@ -265,7 +265,7 @@ Validation splits across two phases: - **At dispatch time** (and at `computeShapes()`), once concrete input shapes are known, we run **shape inference**: the same forward propagation, now carrying each operand's concrete *shape* across the whole graph. We then validate the resulting concrete shapes against every constraint, including buffer sizes. -Some deferral is inherent: a constraint whose truth depends on a concrete value cannot be settled before that value exists. How *much* is deferred is a design choice, and this proposal defers nearly all of it — a dynamic dimension is an opaque name rather than a symbolic expression, so build time can only reject contradictions provable between names ([Symbolic shape expressions](#symbolic-shape-expressions) covers the alternative). So the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so a user agent should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. +Some deferral is inherent: a constraint whose truth depends on a concrete value cannot be settled before that value exists. How *much* is deferred is a design choice, and this proposal defers nearly all of it — a dynamic dimension is an opaque name rather than a symbolic expression, so build time can only reject contradictions provable between names ([Symbolic shape expressions](#symbolic-shape-expressions) covers the alternative). So the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so an implementation should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. The build-time checks are expressed as **three-valued** dimension predicates. Instead of "equal / not-equal", a comparison is *provably-equal*, *provably-unequal*, or *unknown (defer)*. Only a provable contradiction is rejected at build time: @@ -283,15 +283,21 @@ The same predicate is applied uniformly across operators that impose cross-dimen This invariant is one-sided, and only its upper bound is a contract: an implementation **must not** reject a graph that some assignment of concrete input shapes would make valid. Nothing requires it to be good at detecting the rest. The predicate above is the weakest way to honor that — it proves a contradiction only between two static dimensions — and an implementation carrying symbolic *expressions* may reject more, and earlier. Concatenating two `[batch, seq]` tensors along axis 1 doubles the element count, for instance, so reshaping that result back to `[batch, seq]` cannot hold for any input: our model sees only an opaque name for the concatenated axis and defers, while an implementation that knows the axis is `2 * seq` can prove the graph dead and may fail `build()` on it — as some backends performing this analysis internally already do. That is legitimate and costs nothing in portability, since such a graph would fail at every dispatch in any case. It does mean that a successful `build()` is not a portable promise that a graph will ever run — only that this implementation could not prove otherwise. ### Shape computation at dispatch -*Shape computation* is performed by a **shape interpreter** in the user agent, which evaluates a `shape()`-rooted chain down to the concrete integer values a shape parameter needs. Shape inference ([Deferred validation](#deferred-validation)) invokes it at dispatch each time it reaches a `*Dynamic` operator, and uses the values it returns as that operator's output shape. +*Shape computation* is performed on the CPU by a **shape interpreter**, which evaluates a shape chain down to the concrete integer values a shape parameter needs. Shape inference ([Deferred validation](#deferred-validation)) invokes it at dispatch each time it reaches a `*Dynamic` operator, and uses the values it returns as that operator's output shape. + +The shape chains in a graph form a subgraph with a boundary at both ends. Every chain **ends** at an operator whose output shape is determined by the values the chain computes — the `*Dynamic` family, and `range()`, whose length follows from its `start`, `limit` and `delta`. That is what makes it a shape chain, and it is the only way a computed value can influence a shape. Every chain **begins** at a `shape()` output or a build-time constant, and can begin nowhere else, because those are the only values the interpreter can read. Everything in between is ordinary operators evaluated on small integer vectors. + +Two consequences follow, and they are what keep this tractable. A shape can never depend on a value the graph computes from tensor data. And `computeShapes()` is a pure function of the input *shapes*: the same input shapes yield the same output shapes regardless of what the tensors hold. + +Identifying this subgraph does not partition the graph. Its operators remain part of the graph handed to the backend and execute there as usual; the interpreter's evaluation is a separate computation on the CPU that resolves shapes before dispatch. Two questions hide behind "which operators may appear on a shape chain": what a graph may *express*, and what an implementation must be able to *resolve*. -The first is not restricted by operator identity. In principle any operator can compute a shape — `matmul` reducing two 1-D vectors to an element count is a legitimate, if unusual, way to do it. What is restricted is the chain's **root**, described below, and that restriction is what keeps a shape chain from becoming the model. +The first is not restricted by operator identity. In principle any operator can compute a shape — `matmul` reducing two 1-D vectors to an element count is a legitimate, if unusual, way to do it. What is restricted is the chain's **root**, described above, and that restriction is what keeps a shape chain from becoming the model. -Interoperability lives in the second question, and it needs a floor: shapes that resolve in one user agent should resolve in another. ONNX answers the same question with a deliberately narrow closed set — the operators carrying a data-propagation function (`Add`, `Cast`, `Concat`, `Gather`, `Mul`, `Shape`, `Size`, `Slice`, `Squeeze`, `Sub`, `Unsqueeze`) — while ORT's [`symbolic_shape_infer.py`](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py) covers a much wider de-facto set. A required minimum for WebNN belongs between the two, and naming it is work this proposal still owes; the prototype today implements a strict superset of the ONNX set — arithmetic and structural transforms on shape vectors, plus enough floating-point for cases such as `reciprocal`. +Interoperability lives in the second question, and it needs a floor: shapes that resolve in one implementation should resolve in another. ONNX answers the same question with a deliberately narrow closed set — the operators carrying a data-propagation function (`Add`, `Cast`, `Concat`, `Gather`, `Mul`, `Shape`, `Size`, `Slice`, `Squeeze`, `Sub`, `Unsqueeze`) — while ORT's [`symbolic_shape_infer.py`](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py) covers a much wider de-facto set. A required minimum for WebNN belongs between the two. Naming that set is work this proposal still owes, and the intent is that it be normative and extensible — a floor that later revisions can grow, the way the operator set itself grows, rather than a ceiling. The prototype today implements a strict superset of the ONNX set — arithmetic and structural transforms on shape vectors, plus enough floating-point for cases such as `reciprocal`. -Above that floor, a stronger interpreter is a quality-of-implementation matter, with one requirement: an implementation that cannot evaluate a chain must fail with a clean "cannot resolve this shape" error, rather than executing the chain on the accelerator or guessing a value. Heavy operators — `conv2d`, `matmul` on real tensors, attention — are not expected to be resolvable, and an implementation may reject them on a shape chain. Evaluation is integer work on small vectors in the user agent; it is not an execution of the graph, and it never reads a tensor's data. +Above that floor, a stronger interpreter is a quality-of-implementation matter, with one requirement: an implementation that cannot evaluate a chain must fail with a clean "cannot resolve this shape" error, rather than executing the chain on the accelerator or guessing a value. Heavy operators — `conv2d`, `matmul` on real tensors, attention — are not expected to be resolvable, and an implementation may reject them on a shape chain. Evaluation is integer work on small vectors on the CPU, not an execution of the graph — and the properties an implementation would want to rely on, that a chain is small, integral, one-dimensional, and evaluable on the CPU without touching the device, come from the boundary above rather than from the operator list: they hold even for a chain that uses an operator no implementation resolves. A small chain makes this concrete. To reshape a `[1, 'seqlen', 512]` tensor into `[1, 'seqlen', 8, 64]` (splitting the static hidden size into 8 heads × 64) while keeping the dynamic sequence length: @@ -305,7 +311,7 @@ const y = builder.reshapeDynamic(x, newShape); // y: [1, 'seqlen', 8, At dispatch with `seqlen = 37`, the interpreter walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That computed value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. -What the interpreter may read is bounded not by the operators on the chain but by its **root**: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#open-questions) case. +A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#open-questions) case. ### Backends mapping The dimension model maps cleanly onto all three backends, as shown below. The **shape-as-data operator family** is currently implemented only on the ORT backend. @@ -359,7 +365,7 @@ We considered giving each `*Dynamic` operator a shape signature tailored to its ### Symbolic shape expressions A stronger alternative is to carry a symbolic *expression* for every dimension rather than an opaque name, so that build time can prove or refute a graph's shape constraints, report the constraints a model actually requires (divisibility, bounds), and reduce `computeShapes()` to substitution. TensorRT and NNEF's SkriptND both work this way. -We start with opaque names for three reasons. Shape-as-data operators put some shapes beyond any closed-form expression: a target shape is a runtime operand produced by an arbitrary chain, including data-dependent selection such as the `where` used to lower ONNX's `-1`, and a no-axes `squeeze` leaves even the rank data-dependent — so symbolic inference could cover a large subset of graphs but could not replace the dispatch-time gate. Backends cannot consume expressions either: ONNX takes a `dim_param` string, LiteRT a `-1`, Core ML a `RangeDim`, so expressions would remain a user-agent-internal enrichment. And because the graph comes from an untrusted renderer, build-time proving is attacker-influenced work in a privileged process; bounding it by wall-clock time would make build outcomes machine-dependent, so a deterministic budget on expression size and depth would be needed instead. +We start with opaque names for three reasons. Shape-as-data operators put some shapes beyond any closed-form expression: a target shape is a runtime operand produced by an arbitrary chain, including data-dependent selection such as the `where` used to lower ONNX's `-1`, and a no-axes `squeeze` leaves even the rank data-dependent — so symbolic inference could cover a large subset of graphs but could not replace the dispatch-time gate. Backends cannot consume expressions either: ONNX takes a `dim_param` string, LiteRT a `-1`, Core ML a `RangeDim`, so expressions would remain a WebNN-internal enrichment. And because the graph comes from an untrusted renderer, build-time proving is attacker-influenced work in a privileged process; bounding it by wall-clock time would make build outcomes machine-dependent, so a deterministic budget on expression size and depth would be needed instead. Expressions stay attractive as a later, additive layer — for diagnostics and for earlier rejection — and pair naturally with [Bounded (min/max) dimensions](#bounded-minmax-dimensions), which supply the input constraints most build-time proofs need. @@ -385,12 +391,12 @@ The renderer is untrusted, so the service must remain safe on any graph a compro ### Bounded (min/max) dimensions Currently, the proposed model is intentionally unbounded; a dynamic dimension is either provably static or deferred without a specific size range. As a future enhancement, we plan to allow dynamic dimensions to optionally declare a `minSize` and `maxSize` bound (and potentially an "optimal" size). For example: `{name: 'seqlen', minSize: 1, maxSize: 2048}`. -Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. Several runtimes already accept a constraint of this shape and act on it — TensorRT's optimization profiles (min / opt / max), Core ML's `RangeDim`, and OpenVINO's bounded partial shapes — so the hint has somewhere to go rather than terminating in the user agent. +Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. Several runtimes already accept a constraint of this shape and act on it — TensorRT's optimization profiles (min / opt / max), Core ML's `RangeDim`, and OpenVINO's bounded partial shapes — so the hint has somewhere to go rather than stopping at the WebNN layer. ### Shape specialization and preparation Knowing the shapes is not the same as being ready to run them: a backend may still have to plan memory, select kernels, or recompile. What that costs varies a great deal between runtimes: some absorb a shape change almost for free, while others re-compile the graph. So on some backends a caller that changes shape often pays a real price. -`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named a set of concrete shapes. How good a predictor that is depends on why it was called: a framework sizing output tensors for a dispatch it is about to make is one thing, a caller comparing candidate output shapes and running none of them is another, and an implementation cannot distinguish them. The call is also synchronous, which bounds what it can reasonably do there. Even where preparation is warranted it pays off only if the shape is reused, and a user agent has no way to know which shapes are worth keeping ready. One direction worth exploring is to let the caller say so: build with different input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. +`computeShapes()` gives an implementation a natural place to do that preparation, since the caller has just named a set of concrete shapes. How good a predictor that is depends on why it was called: a framework sizing output tensors for a dispatch it is about to make is one thing, a caller comparing candidate output shapes and running none of them is another, and an implementation cannot distinguish them. The call is also synchronous, which bounds what it can reasonably do there. Even where preparation is warranted it pays off only if the shape is reused, and an implementation has no way to know which shapes are worth keeping ready. One direction worth exploring is to let the caller say so: build with different input sizes returning several `MLGraph`s, each bound to a set of concrete shapes and all sharing one copy of the weights. ### Fine-grained Shape Queries Currently, the `shape()` operator returns an operand's entire shape as a 1D tensor. To provide more fine-grained shape retrieval, we may consider introducing two additional operators: