From 895356e2d6216e281e558715b4e09afdfea96a43 Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 5 Sep 2026 10:32:14 -0700 Subject: [PATCH] docs(architecture): align tests with compounding capability --- CLAUDE.md | 4 ++- docs/ANTI_PATTERNS.md | 13 ++++----- docs/architecture-interpretations.md | 12 +++++--- docs/architecture.md | 43 ++++++++++++++++++++-------- docs/learning-flywheel.md | 24 +++++++--------- docs/roadmap-rsi.md | 16 +++++++---- 6 files changed, 67 insertions(+), 45 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 450779959..4f83d62f3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,7 +80,9 @@ Headline entrypoints: `runAgentTask`/`runAgentTaskStream` (`src/run.ts`), the mu ## Self-improvement state & discipline (.evolve/ + the gate) -This repo is the empirical home of the RSI/learning-flywheel thesis, but **mechanism is not evidence**. The binding question is the **gate**: *does any non-blind topology beat blind compute at EQUAL k, under a deployable (non-oracle) selector, on a domain with a correctable middle band, at significant n (paired-bootstrap + BH)?* +The product is an agent system that improves its decisions and learning methods across runs. +Before choosing experiments or deleting mechanisms, read [architecture.md §0.5 and §9](./docs/architecture.md) for success criteria and experiment scope. +Single-run comparisons test specific mechanisms; they do not decide whether capability compounds across projects. **The live science state — every number, what's proven/disproven, the current goal — lives in `.evolve/current.json` + the `memory/` evidence ledger. Read them; do not mirror them here.** `docs/eval-substrate.md` holds the north star (the RSI runtime + its eval substrate) and the measurement non-negotiables. diff --git a/docs/ANTI_PATTERNS.md b/docs/ANTI_PATTERNS.md index b3bf99d30..7d3871f2f 100644 --- a/docs/ANTI_PATTERNS.md +++ b/docs/ANTI_PATTERNS.md @@ -6,15 +6,12 @@ These are repo-level failure modes that have already cost time or produced misleading confidence. If a proposal repeats one, stop and ask what proof would make the work legitimate. -## Mechanism Ahead Of The Gate +## Mechanism Without A Decisive Test -Do not build per-branch adaptive sub-agents, learned planners, corpus promotion, -outer-flywheel machinery, or other high-ceiling mechanisms before a positive -gate result. Expressiveness was the closed gap; evidence is the open one. - -Required proof: a measured non-blind topology beats blind compute at equal k, -under a deployable selector, on a domain with a correctable middle band, with -reported discordant pairs and multiple-comparison discipline. +Build the smallest complete path that can exercise the claimed mechanism before scaling the implementation or experiment. +Use [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope) to define the comparison and the conditions for rejecting it. +A single-run steering result does not decide whether learning across projects works. +Test necessary combinations together, then remove components to identify their contribution. ## Facade Before Substrate Join diff --git a/docs/architecture-interpretations.md b/docs/architecture-interpretations.md index c3666b098..175888a44 100644 --- a/docs/architecture-interpretations.md +++ b/docs/architecture-interpretations.md @@ -178,11 +178,11 @@ The strongest good-faith case: what's wired is the losing half (self-refine) ste ## 5. Gate A — the decision gate for the recursive-driver layer -Build the adaptive driver **only if** this comes back positive: +Test within-run steering with this diagnostic: > On a held-out benchmark, at **equal worker-compute budget** (`k` counts worker ROLLOUTS — each may be a full multi-turn/stateful trajectory, not a single shot), does a **trace + analyst-findings-fed** driver, scored by a **sound non-oracle selector**, beat **blind random@k** selected by that *same* selector — by a statistically significant margin (n large enough for p < 0.05) that **survives test-retest of the selector**? -Until `refine@k-with-findings > random@k at equal compute under a non-oracle selector`, the recursive-driver layer is unjustified overhead and only the minimal honest version (§6) should be built. +Use [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope) for resource accounting, mechanism activation, and the conditions for rejecting the tested design. **Measured: cleared at small n, then RETRACTED to a TIE at power (POWER-16).** On EnterpriseOps-Gym itsm, depth-steered continuation (analyst-fed, `observe()`) beat blind @@ -194,7 +194,9 @@ to a tie when powered, and the program pivoted off this anchor (numbers: codegen** (HumanEval), **positive on stateful agentic domains** with a correctable middle band scored keep-best (EOPS). -**Gate A ≠ project success.** Gate A is the inner GO/NO-GO for *one* component (the within-run driver). The product-success gate is **Gate B** — a positive cross-run score-vs-run slope under a frozen-controller control ([learning-flywheel.md](./learning-flywheel.md)), which is currently **UNMEASURED** (cf. the zero cross-benchmark-transfer admission, §6). A failed Gate A deletes within-run steering; it never bears on Gate B. +**Gate A tests one mechanism under specified conditions.** +Product success is **Gate B**: improvement across runs against an unchanged controller ([learning-flywheel.md](./learning-flywheel.md)). +The historical results above do not establish whether that improvement occurs. --- @@ -206,7 +208,9 @@ The **minimal honest version** survives every critique and yields the proven mor 2. **A deployable, non-oracle selector** scoring each ingest on *trace-observable structural* signal — citation coverage, contradiction-lint pass, staleness, novelty-vs-existing-wiki. This is the missing piece that makes best-of-N actually pay, and it is the same build as landing the *selector ≠ judge* firewall. 3. **The `llm-wiki` maintainer+critic** as the dedup / cite / lint sink (already exists as a skill). -Then run the §5 gate. If a findings-fed driver beats random@k at equal k under that sound selector, the adaptive driver earns its complexity and is built on top — and this surface becomes the first honest validation of the RSI thesis. If not, ship 1+2+3 — agentic RAG with a verifier — and delete the steering machinery with a clear conscience. +Run the §5 diagnostic to assess within-run steering under those conditions. +Apply [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope) before retaining, rejecting, or combining the tested mechanisms. +Test learning across projects separately before claiming that capability compounds. --- diff --git a/docs/architecture.md b/docs/architecture.md index b1d1fa377..a3d45ef7f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -25,6 +25,10 @@ Four claims define the system. The sections below are mechanism in service of these; if a section drifts from one of these, the claim wins and the section is wrong. +The target is a persistent research and engineering system that completes complex software and produces independently checked research. +It can revise its methods, tools, and organization, then retain changes that improve later work. +Recursive self-improvement includes improving how those changes are discovered and tested. + 1. **The atom is a decision, not a spawn.** At every level an agent faces the same question: given the solution so far, the feedback so far, and the budget left, what is the best next move — *keep working · branch · split · get a second opinion · run @@ -34,12 +38,12 @@ if a section drifts from one of these, the claim wins and the section is wrong. AND cheap. Success is **multi-objective**; we do not collapse it to one number until forced to. Today every judge returns a single `score` — that is the **gap to close**, not the design (§6, §5). -3. **Each objective carries its own checker — that is what makes this trainable.** - *Fastest* is graded by a clock, *most secure* by a scanner, *correct* by the tests. - The objective **is** a deployable verifier (§1's *verifier*, distinct from the oracle - and the write-only judge). So the loop has honest, cheap signal at every step, on real - work, **without an answer key** — that is the gift the multi-objective framing buys, - and the reason depth/continuation has something sound to steer on. +3. **Each objective needs evidence with explicit limits.** + Clocks measure elapsed time; tests exercise specified behavior; scanners detect specified security failures. + Passing a check establishes only what that check covers. + Research can require proof checking, independent replication, or new experiments when no answer key exists. + Agents can author working checks, while independent assessment tests the final claims and records unverified assumptions. + The quality and cost of that evidence are part of the research problem. 4. **The improvement that counts is the policy getting better across runs.** Two things improve on two clocks (§2). *Within* a run the **solution** climbs (the artifact gets better round over round). *Across* runs the **decision policy** climbs — it remembers @@ -145,8 +149,8 @@ runs — and *that across-run slope is the success criterion* (**Gate B**, defin trace-fed driver beat a blind same-compute baseline under a non-oracle selector at equal compute* (**Gate A**, defined in [architecture-interpretations.md §5](./architecture-interpretations.md)) — is a separate, -narrower diagnostic; a failed Gate A deletes within-run steering, never the corpus+policy -product. Live results for both gates: `.evolve/current.json` + the memory ledger. +narrower diagnostic governed by the experiment scope in §9. +Live results for both gates: `.evolve/current.json` + the memory ledger. --- @@ -270,7 +274,10 @@ across benchmarks**. Infra is the cost of entry; transfer is the company. --- -## 9. Build order (rung discipline — do not skip) +## 9. Build order and experiment scope + +Choose the smallest complete experiment that can test the claimed advantage. +The mechanisms below are experiment options; an earlier option winning is not a prerequisite for testing a later option. 1. **Atom instance, inference-time.** Driver (`llm-call`, fed by a trace-analyst report) steers a worker over k shots; a **selector** picks the answer @@ -280,13 +287,25 @@ across benchmarks**. Infra is the cost of entry; transfer is the company. one-shot artifact structurally cannot exhibit continuation — the rung-0 "steering loses" result is bench-specific, not domain-general). Status: see the ledger (`.evolve/current.json`). -2. **Escalate the driver to `sandbox-agent` (auto-research)** — only if rung 1 - beats compute-matched random. +2. **Give the driver execution access through `sandbox-agent` (auto-research).** 3. **GEPA** the driver/analyst `context`+prompts, held-out gated. 4. **Composition lift** — `fork`/coordinator/nested (driver-of-drivers). 5. **Cross-benchmark transfer** — one learned controller, many benchmarks. The moat. -Each rung must beat compute-matched random before the next is funded. +Test combinations when the claimed benefit depends on interactions between delegation, retained knowledge, continued work, or learned decisions. +Remove components in controlled comparisons to identify their contribution. +An isolated component losing does not establish that the combination cannot help. + +Measure the hardest independently checked work completed, results across resource budgets, and improvement across successive projects. +Use equal actual resources within each comparison and keep model versions and task difficulty controlled. +Report learning costs separately and include them in total costs over the declared project horizon. + +Reject a mechanism only within the conditions tested. +The task must exercise the conditions where an advantage is claimed. +The mechanism must run observably, and the assessment must detect the smallest useful effect. +Use enough paired cases and a declared stopping rule to support the decision. +When those conditions hold and the effect is below the useful threshold, reject or simplify that mechanism for those conditions. +If a test condition is missing, record what remains untested and name the missing condition before further work. --- diff --git a/docs/learning-flywheel.md b/docs/learning-flywheel.md index f256b4e17..909b58fbf 100644 --- a/docs/learning-flywheel.md +++ b/docs/learning-flywheel.md @@ -36,16 +36,14 @@ > corpus, the gain survives a **frozen-controller control** (re-running an earlier controller shows > no slope), it is significant at adequate n (paired-bootstrap + BH), and it is graded by a > **deployable checker** — never the answer oracle or the write-only judge. *Multi-objective* is -> load-bearing: success is a vector (correct · fast · secure · cheap), and **each objective ships -> its own deployable checker** — tests, a clock, a scanner, a cost meter. That is what gives the -> flywheel honest, cheap, oracle-free signal on real work at every step (and what depth/continuation -> steers on). This OUTER-loop slope is THE success criterion. The +> load-bearing: success is a vector (correct · fast · secure · cheap), with evidence scoped to each objective. +> Tests, clocks, scanners, and cost meters provide partial measurements; record each check's coverage and unverified assumptions. +> This OUTER-loop slope is THE success criterion. The > within-run "trace+findings-fed controller beats the blind same-compute baseline under a non-oracle > selector at **equal compute**" question is a separate, narrower diagnostic — **Gate A**, the -> GO/NO-GO for building the recursive-driver layer (see [roadmap-rsi.md](./roadmap-rsi.md)). A failed -> Gate A deletes within-run steering only; it never touches this corpus+controller product. Equal -> compute (Σ rollouts × turns per arm) is the anti-confound guard in both gates and is **silent on -> statefulness**: the budget may be spent as one deep trajectory, K shallow attempts, or any mix. +> comparison for within-run steering, scoped by [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope). +> Compare actual resource use in both tests, including learning costs over the declared project horizon. +> The budget may fund one deep trajectory, several shallow attempts, or a mixture. ## The flywheel @@ -111,9 +109,8 @@ It is real (not vapor) only under three constraints: is *why* the integrity rule (judge never feeds steering/selection) is non-negotiable. 2. **Shared corpus `D`.** Improvements persist and are evidenced *across* levels — a level-1 gain shows up in the corpus the level-0 runs produced. -3. **Rung-by-rung earning.** Level *n* is real iff it **measurably lifts level n−1 on `J`**. - Recursion is real ∝ rungs earned; skip a rung and you stack noise on noise. (We are at L0 - with ~0 confirmed signal — so the stack is the north star, built strictly bottom-up.) +3. **Evidence per learning level.** Level *n* contributes only when it **measurably lifts level n−1 on `J`**. + This requirement concerns evidence; construction and comparison follow [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope). **This subsumes everything in this repo and this design:** the worker, the `f(trace)` steer, the controller-as-signatures, GEPA, `meta-harness`, AND the **skill-governor** (which skill to @@ -229,9 +226,8 @@ A separate final-test partition is required because source labels alone cannot p - **Eval economics is the moonshot bottleneck, not controller cleverness.** Build the offline corpus/replay so search is affordable. Don't build the optimizer cathedral over a metric you can only sample a few hundred times with overlapping CIs. -- **Prove signal per rung before escalating cost.** random → fixed → `LLM(trace)` → - agentic-driver. Each rung must beat *compute-matched* random before the next is justified. - Don't jump to the unbounded agentic driver to (expensively) re-derive that more-compute ≈ 0. +- **Choose a decisive test before escalating cost.** + Follow [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope) for complete mechanisms, combinations, resource accounting, and rejection conditions. ## Honest status (updated 2026-06-10) diff --git a/docs/roadmap-rsi.md b/docs/roadmap-rsi.md index 8d25c1ef7..7c3f723a2 100644 --- a/docs/roadmap-rsi.md +++ b/docs/roadmap-rsi.md @@ -4,13 +4,16 @@ Companion to [architecture.md](./architecture.md) (the spine) and [architecture- ## The principle: make it measurable before you build it -Building the recursive-driver layer is gated on **Gate A** (the inner GO/NO-GO) from [architecture-interpretations.md §5](./architecture-interpretations.md#5-the-decision-gate): +**Gate A** tests within-run steering ([architecture-interpretations.md §5](./architecture-interpretations.md#5-gate-a--the-decision-gate-for-the-recursive-driver-layer)): -> Build the adaptive driver only if, at **equal worker-compute** (Σ rollouts × turns — `k` counts ROLLOUTS, each of which may be a full multi-turn/stateful trajectory), a **trace+findings-fed** driver scored by a **sound non-oracle selector** beats **random@k** selected by that *same* selector — significantly, and surviving selector test-retest. +> Does a trace-informed driver beat random attempts under the same answer selection method, at equal actual resources, with enough evidence to distinguish useful improvement? -**Gate A is NOT the project-success criterion.** Project success is **Gate B** — the cross-run flywheel slope on a **multi-objective** (correct · fast · secure · cheap, each its own deployable checker), verifier-graded score ([learning-flywheel.md](./learning-flywheel.md), [architecture.md §0.5](./architecture.md)). Gate A only decides whether the *within-run adaptive driver* is worth building; a failed Gate A deletes within-run steering, never the corpus+controller product. **Note the asymmetry in this roadmap:** Phases 0–4 below are all instrumented for Gate A (within-run, single-objective correctness) — that is the cheap, buildable diagnostic. Gate B (the across-run policy curve on a multi-objective task stream) is the actual success criterion and is **not yet instrumented**; standing it up is the durable next step, not a replay over the existing single-objective corpus. +Project success is **Gate B**: improvement across projects under the criteria in [architecture.md §0.5](./architecture.md#05-what-we-are-building-and-what-better-means-the-four-claims). +Apply [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope) before using Gate A to reject a mechanism or choose another experiment. +The phase map below records the within-run investigation; it does not make that investigation a prerequisite for testing learning across projects. -So the phases are ordered to make each step measurable on an honest baseline *before* the next is built. Build order: **honest baseline → the cheap win (selector) → wire the intelligence (analyses) → grow the language (ISA) → the use case (acquisition)**. The cleanup and doc tracks run in parallel because they are additive-safe. +The phases organize that investigation around measurable behavior. +Use the canonical experiment scope when choosing a different order or testing interacting mechanisms together. > **Status (updated POWER-16).** The canonical "drive an agent" path is the **agent-driver**: > an `AgentProfile` driving another via `createCoordinationTools` @@ -41,7 +44,7 @@ So the phases are ordered to make each step measurable on an honest baseline *be |---|---|---|---|---|---| | **0** | Honest baseline + preconditions (no kernel change) | — | Every runner reports `random@k` at equal k; corpus has a measurable discordant-pair rate | low | **done** — `runPool` landed (`bench/src/run-pool.ts`); the corpus + `corpus-report.mts` BH-FDR path is the `random@k`-control measurement surface | | **1** | Deployable non-oracle selector | 0 | `selector@k > random@k` significant (paired bootstrap + BH), low test-retest flip rate, on a frozen held-out split | low–med | **built + measured** — verifier-grounded selector positive on HumanEval (+12pp verifier−sc CI [+4,+22] / +18pp random−blind, BH-sig, n=50 k=4); answer-agreement negative (finsearch −8.2pp, aec −9.4pp) | -| **2** | Wire `analyses → driver` (the missing edge) | 0, 1 | **Gate A** (inner GO/NO-GO for the recursive-driver layer): `refine@k-with-findings > random@k` at equal compute under the Phase-1 selector, significant, survives test-retest — NOT flywheel success (Gate B) | med | the diagnosis→steer edge lives on the agent-driver (`observe()` → `createCoordinationTools`); Gate A itself **ran on the Supervisor substrate, then RETRACTED to a tie at power** (header note) | +| **2** | Wire `analyses → driver` (the missing edge) | 0, 1 | **Gate A**: compare within-run steering under the [canonical experiment scope](./architecture.md#9-build-order-and-experiment-scope) | med | the diagnosis→steer edge lives on the agent-driver (`observe()` → `createCoordinationTools`); Gate A itself **ran on the Supervisor substrate, then RETRACTED to a tie at power** (header note) | | **3** | Grow the ISA (`select` then `seq`) | 2 | A strategy expressing `select`/`seq` beats a flat one on the same harness | med (3a) / high (3b) | **superseded** — `defineStrategy` (`src/runtime/strategy.ts`) is the richer program space: a strategy is ordinary code with arbitrary sequencing and branching | | **4** | Acquisition adapter (research use case) | 0, 1 (parallel to 2) | Active acquisition beats random acquisition on the deployable coverage-vs-budget curve under a *structural* gap signal | med–high | open | @@ -74,7 +77,8 @@ At audit time the selector was **faked with the judge**: `defaultSelectWinner` ( The load-bearing edge. **Status: lives on the agent-driver.** The diagnosis→decision edge runs on the **agent-driver**: a parent `AgentProfile` consumes `observe()` findings (`AnalystFinding`, the substrate type from `@tangle-network/agent-eval` — **never redefined**, the layering rule) and steers its child via `createCoordinationTools` (`src/mcp/tools/coordination.ts`) over the `Scope`/`Supervisor`. The `runAgentRounds` kernel (`src/runtime/run-loop.ts`) stays analyst-free. **No bench feeds the findings-fed treatment arm against the `random@k` control under the Phase-1 selector live yet** — that is the remaining work on this substrate. -**Exit gate — Gate A (inner GO/NO-GO).** `refine@k-with-findings > random@k` at equal compute under the Phase-1 selector, statistically significant, surviving selector test-retest. **If it fails:** stop building the *within-run recursive-driver layer* — ship Phases 0–1 + Phase 4 (agentic RAG with a verifier) and delete the *steering machinery*. The recursive-driver layer is unjustified overhead unless this clears. **This is scoped to within-run steering only — it is NOT the flywheel-success criterion (Gate B, [learning-flywheel.md](./learning-flywheel.md)); a failed Gate A never deletes the corpus+controller product.** +**Exit criterion — Gate A.** `refine@k-with-findings > random@k` at equal actual resources under the Phase-1 selector, with statistical support and repeatable selection. +Apply [architecture.md §9](./architecture.md#9-build-order-and-experiment-scope) when interpreting a result below the useful threshold. **Gate A status: TIE at power (POWER-16), on the `Scope`/`Supervisor` substrate** — the n=16 "+16.4pp cleared" signal (depth-steered continuation, analyst-fed via `observe()`, vs blind breadth at equal compute under keep-best scoring) collapsed to depth−breadth +4.7pp CI [−1.9, +11.4] at n=48 (header note). At most a small effect, not a cleared keystone; the program pivoted off it.