Skip to content

fix(learning): preserve optimizer selection and measurement integrity - #738

Merged
drewstone merged 1 commit into
mainfrom
audit/learning-system-20260905
Sep 5, 2026
Merged

fix(learning): preserve optimizer selection and measurement integrity#738
drewstone merged 1 commit into
mainfrom
audit/learning-system-20260905

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Complete optimization methods could lose their selected winner when selfImprove() ran them through the native proposal loop.
An unchanged selection failed as a duplicate candidate; another selected surface could be replaced by the baseline using pooled training scores.
The method now owns search and selection, and a shared final comparison measures its selected surface without executing training again.

The change also repairs measurement and accounting failures:

  • Bind native search, final comparisons, and defineAgentEval.evaluate() caches to surface content and execution revision.
  • Require the complete evaluator manifest for a premeasured baseline, including judge revision.
  • Require every final case and replica before comparing methods.
  • Reconcile reported optimizer spending with method-attributed receipts, including concurrent methods, incomplete usage, and floating-point sums.
  • Count a shared unchanged campaign once in result analysis.
  • Report native candidate uncertainty as absent when it was not estimated.
  • Feed observed behavior-cell variance into the next exploration allocation.

The complete method path retains actual method output and canonical measurement provenance.
The predictive-validity researcher documentation now describes its implemented recommendation behavior.
Process tests wait for child readiness and await cleanup; real-Git fixtures have an explicit startup allowance.

API migration in 0.174.0

SelfImproveResult is a union of SelfImproveProposerResult and SelfImproveMethodResult.
Use mode to narrow before reading native generations.
Method results expose raw.method, combined cost, receipt-only ledgerCost, and tangle.method-improvement provenance.
Deferred method results have null final scores and no lift.
Native-only search controls are rejected in method mode.
Standalone premeasured baselines must use surfaceDispatchRef(baselineSurface, dispatchRef) and the same judge revision.
The examples and campaign documentation cover these contracts.

Validation

  • Full suite at four workers:5,763 passed,3 skipped,0 failed across394 files in140.77s.
  • Focused process, comparison, and complete-method integrity tests:185 passed.
  • Package typecheck, example typecheck, script typecheck, lint, build, and packed-package verification passed.
  • Regression fixtures execute the real campaign, cost ledger, process runner, and analysis paths with deterministic agent callbacks.
    They make no paid provider calls and do not measure current model quality.

The shared ledger enforces limits on admitted calls.
Custom methods remain responsible for arbitrary external work they do not report.

@drewstone
drewstone merged commit 0692922 into main Sep 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant