W-23599769: benchmark existing artifacts and CLI commands

.gitignore

@@ -29,5 +29,6 @@ grimoires/
 
 # Superpowers SDD scratch workspace (briefs, reports, review packages, ledger)
 .superpowers/
-# Superpowers implementation plans — kept local, not committed (specs are committed)
-docs/superpowers/plans/
+# Superpowers implementation plans are local scratch artifacts, never commit them.
+/docs/superpowers/plans/
+/docs/superpowers/plans/**/*.md

AGENTS.md

@@ -176,7 +176,9 @@ Benchmarks are opt-in: `./gradlew benchmarkCompare -Pbenchmark=true`. Runner tas
 `ext.benchmarkRunner = true`; the root aggregator auto-discovers them. New runners must
 use the shared corpus/schema and write `benchmarks/results/<runner>-<timestamp>.json`.
 Do not add benchmark execution to normal `build` or `test`. The CLI benchmark requires a
-binary built with `-Pbenchmark=true`; `DW_BENCH_BIN` may select a prebuilt one.
+normal `dw` binary. `-Pbenchmark=true` gates Gradle benchmark task execution; it does not
+make the artifact benchmark-capable. `DW_BENCH_BIN` selects an existing ordinary `dw`
+binary.
 
 CI builds Ubuntu and Windows with GraalVM 24. Native CLI regression suites and Node TCK
 run only on `master`. Run the smallest relevant suite, then the nearest module test; use

benchmarks/README.md

@@ -13,12 +13,8 @@ Language-agnostic benchmark harness for the DataWeave native-lib wrappers.
   (Scala/Gradle subproject `:benchmarks-engine`, depends on `org.mule.weave:runtime` at
   the same `weaveVersion` the native image is built from). `runners/python/` is the
   Python runner (stdlib scripts under `native-lib`, wrapping the same staged `dwlib` as Node).
-  `runners/cli/` is the CLI runner: a Node parent that spawns the `dw` native
-  binary (built with `-Pbenchmark=true`, which compiles in an in-binary
-  benchmark harness gated by `BenchmarkMode.ENABLED` and dispatched via the
-  `DW_BENCH` env var — the shipped `dw` contains none of it). It emits
-  `cold-start`, `first-run`, and `warm`; it does **not** emit `streaming`
-  (the `dw run` path has no chunked-input FFI like the library's).
+   `runners/cli/` is the CLI runner: a Node parent that spawns normal `dw run`
+   commands and emits only end-to-end `first-run` measurements.
 - `report/report.mjs` — joins result files against the manifest and prints a comparison table.
 - `results/` — gitignored per-run output.
 
@@ -27,14 +23,20 @@ Language-agnostic benchmark harness for the DataWeave native-lib wrappers.
 `cold-start` and `first-run` (fresh process per sample), `warm` (in-process steady state),
 `streaming` (MB/s). Each case declares which apply via `metrics[]`.
 
-**Cold-start is measured by the parent, not the child** — every runner spawns a fresh
-child that prints a `READY` marker the instant its runtime is initialized, and the parent
-records wall-clock from just-before-spawn to that marker. So cold-start includes process
-launch + library/class load + runtime init on all three runners, which is what makes the
-native-image-vs-JVM comparison meaningful (the native image has no JVM to boot; the JVM's
-cold cost *is* launch + classload). Adding a runner requires the same protocol: print
-`READY` (flushed) after init, then a JSON line with the in-process `firstRunMs`. Note only
-the first sample sees a truly cold OS page cache; the reported median is warm-cache init.
+For the CLI runner, `first-run` is end-to-end `dw run` command latency. Other
+runners' `first-run` is in-process compile-and-execute latency. The CLI emits
+no `cold-start`, `warm`, or `streaming` rows, so its table deltas remain visible
+but qualify a different measurement boundary.
+
+**Cold-start is measured by the parent, not the child** for the Node, Python, and engine
+runners. Their fresh child prints a `READY` marker the instant its runtime is initialized,
+and the parent records wall-clock from just-before-spawn to that marker. Cold-start therefore
+includes process launch + library/class load + runtime init, which makes the native-image-vs-JVM
+comparison meaningful (the native image has no JVM to boot; the JVM's cold cost *is* launch +
+classload). These in-process runners use the `READY` (flushed) plus JSON `firstRunMs` protocol.
+The CLI does not use that protocol: it measures each normal `dw run` process from spawn to
+successful exit. Only the first sample sees a truly cold OS page cache; the reported median is
+warm-cache init.
 
 ## Prerequisites
 
@@ -45,19 +47,46 @@ and `JAVA_HOME` set to it (see the root README / `CLAUDE.md`). The pinned build
 `graalvmVersion` in `gradle.properties`. The **engine runner alone** drives the JVM
 `DataWeaveScriptingEngine` and runs on any JDK — no native image required.
 
-The **CLI runner** requires the bench-enabled binary
-(`./gradlew native-cli:nativeCompile -Pbenchmark=true`); set `DW_BENCH_BIN` to
-point at a prebuilt one. Like the library runners it needs the GraalVM toolchain.
+`DW_BENCH_BIN` points to an ordinary prebuilt `dw` binary; the CLI runner does
+not build it when the override is supplied.
 
 ## Running
 
 The one-shot cross-runner comparison — runs **every** registered runner and prints the table:
 
     ./gradlew benchmarkCompare -Pbenchmark=true          # all runners + comparison report
 
+### Running against pre-built wrapper artifacts
+
+The Node and Python runners can benchmark pre-built wrapper artifacts via env vars, skipping
+their corresponding local wrapper build or staging task:
+
+- **`DW_BENCH_NODE_PACKAGE`** — absolute path to an extracted `@dataweave/native` package
+  directory (must contain `dist/index.js`). Example:
+
+      DW_BENCH_NODE_PACKAGE=/tmp/artifacts/node/package \
+        ./gradlew native-lib:benchmarkNode -Pbenchmark=true
+
+- **`DW_BENCH_PY_SITE`** — absolute path to a site-packages-style directory containing
+  `dataweave/__init__.py`. Populate with `pip install --target <dir> <wheel>`. Example:
+
+      pip install --target /tmp/artifacts/py dataweave-0.0.1-py3-none-any.whl
+      DW_BENCH_PY_SITE=/tmp/artifacts/py \
+        ./gradlew native-lib:benchmarkPython -Pbenchmark=true
+
+If the env var is set but the target is invalid, the runner fails immediately rather than
+falling back to the source tree. For a cross-runner comparison with both wrapper overrides:
+
+    DW_BENCH_NODE_PACKAGE=/tmp/artifacts/node/package \
+      DW_BENCH_PY_SITE=/tmp/artifacts/py \
+      ./gradlew benchmarkCompare -Pbenchmark=true
+
+Use `DW_BENCH_BIN` to point the CLI runner at an ordinary prebuilt `dw` binary;
+when it is set, `benchmarkCli` does not run a local `nativeCompile`.
+
 Single-runner options:
 
-    ./gradlew native-lib:benchmark -Pbenchmark=true              # Node only: build wrapper, run, report
+    ./gradlew native-lib:benchmarkNode -Pbenchmark=true          # Node only: writes results/node-<ts>.json
     ./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true # engine (JVM) only: writes results/engine-<ts>.json
     ./gradlew native-lib:benchmarkPython -Pbenchmark=true        # Python only: writes results/python-<ts>.json
     ./gradlew native-cli:benchmarkCli -Pbenchmark=true            # CLI only: writes results/cli-<ts>.json
@@ -68,6 +97,11 @@ Or directly, once the wrapper is built (`./gradlew native-lib:buildNodePackage`)
     node runners/node/emit.mjs                            # writes results/node-<ts>.json
     node report/report.mjs results/*.json                 # renders the table
 
+`gen-inputs.mjs` reuses an existing `corpus/inputs/generated/records-large.json` so
+every runner in a comparison uses the same bytes. `BENCH_LARGE_N` is applied only when
+the file is first generated; delete `corpus/inputs/generated/records-large.json` before
+running the generator to create a corpus with a different record count.
+
 Results (`results/*.json`) are local-only and gitignored; no history is accumulated (see the
 design spec). To publish a snapshot, render a self-contained Markdown report with charts:
 

benchmarks/corpus/gen-inputs.mjs

@@ -1,6 +1,6 @@
 // Deterministically regenerate large inputs. No randomness -> comparable across
 // machines and runners. Size overridable via BENCH_LARGE_N (default 50000).
-import { writeFileSync, mkdirSync } from "node:fs";
+import { existsSync, statSync, writeFileSync, mkdirSync } from "node:fs";
 import { join, dirname } from "node:path";
 import { fileURLToPath } from "node:url";
 
@@ -9,10 +9,15 @@ const outDir = join(__dirname, "inputs", "generated");
 mkdirSync(outDir, { recursive: true });
 
 const n = Number(process.env.BENCH_LARGE_N ?? 50000);
+const path = join(outDir, "records-large.json");
+if (existsSync(path)) {
+  console.log(`reusing ${statSync(path).size} byte input at ${path}`);
+  process.exit(0);
+}
+
 const records = [];
 for (let i = 1; i <= n; i++) {
   records.push({ id: i, name: `item_${i}`, value: i * 3 });
 }
-const path = join(outDir, "records-large.json");
 writeFileSync(path, JSON.stringify(records));
 console.log(`wrote ${n} records to ${path}`);

benchmarks/lib/env.mjs

@@ -24,12 +24,15 @@ function readCommit() {
   }
 }
 
-// Best-effort identity of the staged dwlib: first 8 hex of a sha256 over
+// Best-effort identity of the selected dwlib: first 8 hex of a sha256 over
 // (size + first 64KB). Cheap, stable, and enough to detect a lib swap.
-function readDwlibBuildId() {
-  const base = join(REPO_ROOT, "native-lib", "node", "native");
-  for (const ext of [".dylib", ".so", ".dll"]) {
-    const p = join(base, `dwlib${ext}`);
+function readDwlibBuildId(dwlibPath) {
+  const paths = dwlibPath && existsSync(dwlibPath)
+    ? [dwlibPath]
+    : [".dylib", ".so", ".dll"].map((ext) => {
+      return join(REPO_ROOT, "native-lib", "node", "native", `dwlib${ext}`);
+    });
+  for (const p of paths) {
     if (existsSync(p)) {
       const buf = readFileSync(p).subarray(0, 65536);
       const size = statSync(p).size;
@@ -40,9 +43,9 @@ function readDwlibBuildId() {
 }
 
 /**
- * @param {{runner:string, runtimeVersion:string}} opts
+ * @param {{runner:string, runtimeVersion:string, dwlibPath?:string}} opts
  */
-export function gatherEnv({ runner, runtimeVersion }) {
+export function gatherEnv({ runner, runtimeVersion, dwlibPath }) {
   const cpus = os.cpus();
   return {
     runner,
@@ -51,6 +54,6 @@ export function gatherEnv({ runner, runtimeVersion }) {
     runtimeVersion,
     weaveVersion: readWeaveVersion(),
     commit: readCommit(),
-    dwlibBuildId: readDwlibBuildId(),
+    dwlibBuildId: readDwlibBuildId(dwlibPath),
   };
 }

benchmarks/lib/env.test.mjs

@@ -1,7 +1,25 @@
 import { test } from "node:test";
 import assert from "node:assert/strict";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { createHash } from "node:crypto";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
 import { gatherEnv } from "./env.mjs";
 
+const tempDirs = [];
+
+function makeTempDir() {
+  const dir = mkdtempSync(join(tmpdir(), "dw-bench-env-test-"));
+  tempDirs.push(dir);
+  return dir;
+}
+
+test.after(() => {
+  for (const dir of tempDirs) {
+    rmSync(dir, { recursive: true, force: true });
+  }
+});
+
 test("gatherEnv returns all required fields", () => {
   const env = gatherEnv({ runner: "node-wrapper", runtimeVersion: "node vX" });
   for (const key of ["os", "cpu", "runtimeVersion", "weaveVersion", "commit", "dwlibBuildId"]) {
@@ -14,3 +32,22 @@ test("gatherEnv reads the pinned weaveVersion from gradle.properties", () => {
   // gradle.properties pins e.g. 2.12.0-YYYYMMDD; assert it looks like a weave version.
   assert.match(env.weaveVersion, /^\d+\.\d+\.\d+/);
 });
+
+test("gatherEnv attributes an explicitly selected native library", () => {
+  const libraryPath = join(makeTempDir(), "dwlib.dylib");
+  const libraryBytes = Buffer.from("external native library fixture");
+  writeFileSync(libraryPath, libraryBytes);
+  const expectedBuildId = "dwlib-" + createHash("sha256")
+    .update(String(libraryBytes.length))
+    .update(libraryBytes.subarray(0, 65536))
+    .digest("hex")
+    .slice(0, 8);
+
+  const env = gatherEnv({
+    runner: "node-wrapper",
+    runtimeVersion: "node vX",
+    dwlibPath: libraryPath,
+  });
+
+  assert.equal(env.dwlibBuildId, expectedBuildId);
+});

benchmarks/report/report.mjs

@@ -150,6 +150,12 @@ export function renderMermaidCharts(table) {
   return blocks.join("\n\n");
 }
 
+export function renderMetricNotes(results) {
+  if (!results.some((result) => result.runner === "cli")) return "";
+  return "CLI `first-run` is end-to-end `dw run` command latency. " +
+    "Other runners' `first-run` is in-process compile-and-execute latency.";
+}
+
 /**
  * A self-contained Markdown report: provenance (commit + date), the numeric
  * table, then a Mermaid bar chart per (case, metric) — one bar per runner.
@@ -168,6 +174,8 @@ export function renderMarkdown(table, results, { baselineRunner, stamp }) {
     "> Indicative only — timings are from a single run on one machine, not a dedicated bench box.",
     ""
   );
+  const metricNotes = renderMetricNotes(results);
+  if (metricNotes) out.push(metricNotes, "");
 
   out.push("## Table", "");
   out.push("| " + table.header.join(" | ") + " |");
@@ -214,6 +222,11 @@ export function main(argv) {
     console.log(`⚠️  WEAVE VERSION SKEW: comparing across ${skew.join(" vs ")} — deltas are not clean.`);
     console.log("");
   }
+  const metricNotes = renderMetricNotes(results);
+  if (metricNotes) {
+    console.log(metricNotes);
+    console.log("");
+  }
 
   const table = buildTable(manifest, results, baselineRunner);
   const { header, rows, otherRunners } = table;

benchmarks/report/report.test.mjs

@@ -11,6 +11,7 @@ import {
   buildTable,
   dedupeLatestByRunner,
   renderMermaidCharts,
+  renderMetricNotes,
   renderMarkdown,
 } from "./report.mjs";
 
@@ -82,6 +83,33 @@ test("renderMarkdown emits no streaming non-comparable footnote", () => {
   });
   assert.ok(md.includes("| map-scale | streaming | MB/s |"), "streaming row is present");
   assert.ok(!md.includes("not like-for-like across runners"), "footnote removed");
+  assert.equal(renderMetricNotes(results), "", "metric note is omitted without CLI results");
+});
+
+test("report rendering labels CLI first-run as end-to-end and other runners as in-process", () => {
+  const manifest = loadManifest(CORPUS);
+  const engine = load("engine-b.json");
+  const cli = {
+    ...engine,
+    runner: "cli",
+    cases: engine.cases
+      .filter((result) => result.metric === "first-run")
+      .map((result) => ({ ...result })),
+  };
+  const results = [engine, cli];
+  const table = buildTable(manifest, results, "engine");
+  const md = renderMarkdown(table, results, {
+    baselineRunner: "engine",
+    stamp: { commit: "abc1234", date: "2026-08-10T14:33:03Z" },
+  });
+
+  assert.ok(md.includes("CLI `first-run` is end-to-end `dw run` command latency."));
+  assert.ok(md.includes("Other runners' `first-run` is in-process compile-and-execute latency."));
+  assert.equal(
+    renderMetricNotes(results),
+    "CLI `first-run` is end-to-end `dw run` command latency. " +
+      "Other runners' `first-run` is in-process compile-and-execute latency."
+  );
 });
 
 test("renderMermaidCharts emits one chart per (case, metric) with a bar per runner", () => {