Skip to content

warn when the model does not start where the user asked - #252

Open
jdeast wants to merge 1 commit into
masterfrom
worktree-user-start-check
Open

warn when the model does not start where the user asked#252
jdeast wants to merge 1 commit into
masterfrom
worktree-user-start-check

Conversation

@jdeast

@jdeast jdeast commented Sep 11, 2026

Copy link
Copy Markdown
Owner

When a user sets a value, the model should produce that value or say why it can't. The second half did not exist, and the failure was silent: examples/ob09020 pinned mulensevent.t_E = 76.9 and started at 74.48, and the only way to find that out was to compile the graph by hand.

Measured across the shipped examples: 19 build, 14 user values are not reproduced.

The check

ModelAuditor.check_user_starts() reads every user-set initval off the compiled graph — not p.value (a draw from the prior) and not p.initval (the ledger, which is half of what is under test). Both look right while the model starts somewhere else. inspect_start prints the misses after the startup table, beside the existing flat/unused warnings.

Two reasons, separated by the ledger rather than guessed:

ledger vs user built vs user reported as
same differs approximate — the derivation cannot preserve it
differs overspecified — and _last_solved_by names the equation

Calling the first "overspecified" would send someone hunting for a conflicting pin that does not exist, so the distinction is load-bearing rather than cosmetic.

What the sweep found

All 14 misses are the first kind. Nothing in any shipped example is being trampled — every one is a value the engine faithfully kept and the derivation could not rebuild. That is direct evidence the precedence system is not the problem here, which is the opposite of where this investigation started.

The error tracks pi_rel exactly as the heliocentric/geocentric diagnosis predicts: ob09020 (pi_rel = 1.21 mas, a 747 pc lens) is ~10x the bulge events at pi_rel ~ 0.1. Two finds that were previously invisible: gj1214's transit.MIRILRS.jitter_variance (+49.5%) and galactic_model's star.BulgeTarget.mass, which starts at exactly 10**-0.3 because a mass pin never reaches logmass.

Two traps this walked into, both now regression-tested

Degrees are periodic. The prototype reported four examples as broken over bigomega/alpha pairs exactly 360 apart (352.57 vs -7.43, 210 vs -150, ...). Those are the same start. A warning block with false positives is one nobody reads.

The ledger is in INTERNAL units (config.py:890) while the params file is in user units, so comparing them raw calls every converted parameter overspecified — star.ra holds 4.6095 rad against a written 264.105 deg. The first draft did exactly that. test_the_ledger_is_compared_in_user_units fails on the pre-fix code; that was verified by mutation, not assumed.

examples/ob09020 back to published values

A params file is a template, and a reader cannot tell a transcribed number from a fitted one when every entry carries a paragraph defending it. The justifying prose is gone. pi_E_N returns to Skowron's -0.022 from a hand construction, q to the printed 0.273, and theta_E = 2.95 is now pinned.

The set does not close — marginal medians from one table plus physical parameters from another paper — and that is the normal case, not a fault. The new report says so instead of the file hiding it behind tuned numbers.

bigomega is the one entry that cannot be "the published value": Yee's Omega_node = -7.767 is in their frame, and the documented mapping consumes phi_pi, which is a model output. Re-converting against the delivered phi_pi (162.068, from 158.795) gives 340.755; the same rule reproduces the old 337.484 to 0.002 deg. Start logp +19,024 -> +36,397, with alpha_0 = 189.082 (published 189.08) and s_0 = 0.4264 (0.4294) unchanged — the published-only template starts better than the hand-tuned one did.

Measured, because it decides whether this is seedable at all: this fit cannot walk in bigomega. +/-10 deg costs >1e6 nats, 60 deg costs 7e5, and the midpoint is worse than either end — a dead mode, not a slow one, against PTDE's factor-200 flattening.

Verification

8 new unit tests. 85 passed across the acceptance, keplerian, polish, hot-chains and wrapup-guard files. The ob09020 acceptance fixture is regenerated for the deliberate seed change (81 terms); the other 12 stay bit-identical. Pre-push hook green (ruff, ruff format, full suite).

Follow-up

The heliocentric/geocentric frame fix removes the largest remaining class of misses. It is now an improvement rather than a prerequisite, since the contract is satisfied by the warning.

🤖 Generated with Claude Code

THE CONTRACT: when a user sets a value, the model produces that value or
says why it cannot.  The second half did not exist, and the failure was
SILENT -- examples/ob09020 pinned mulensevent.t_E = 76.9 and started at
74.48, and the only way to discover that was to compile the graph by
hand.  Measured across the shipped examples: 19 build, 14 user values are
not reproduced.

ModelAuditor.check_user_starts() reads every user-set initval off the
COMPILED GRAPH -- not p.value (a draw from the prior) and not p.initval
(the ledger, which is half of what is under test), both of which look
right while the model starts elsewhere.  inspect_start prints the misses
after the startup table, beside the existing flat/unused warnings.

Two reasons, separated by the ledger rather than guessed:

  ledger == user, built != user -> the DERIVATION cannot preserve it.
      ob09020's t_E: recorded at rank 100, never overwritten, and still
      3.15% out because the seed path is heliocentric while the graph is
      geocentric (symbolic_physics.py's SEEDING APPROXIMATION).
  ledger != user -> OVERSPECIFIED, and _last_solved_by names the equation
      that overwrote it.

Calling the first "overspecified" would send the user hunting for a
conflicting pin that does not exist, so the distinction is load-bearing.

WHAT THE SWEEP FOUND.  All 14 misses are the first kind; nothing in any
shipped example is being trampled, which is direct evidence the
precedence system is not the problem here.  The error tracks pi_rel
exactly as the frame diagnosis predicts: ob09020 (pi_rel = 1.21 mas, a
747 pc lens) is ~10x the bulge events at pi_rel ~ 0.1.  Two finds that
were previously invisible: gj1214's transit.MIRILRS.jitter_variance
(+49.5%) and galactic_model's star.BulgeTarget.mass, which starts at
10**-0.3 because a mass pin does not reach logmass.

Degrees are wrapped: the prototype reported four examples as broken over
bigomega/alpha pairs exactly 360 apart, and a warning block with false
positives is one nobody reads.  The ledger is converted through the
Parameter before comparison -- it stores INTERNAL units (config.py:890)
while the file is in user units, so a raw compare calls every converted
parameter overspecified (star.ra: 4.6095 rad against a written 264.105
deg).  The first draft did exactly that; the regression test fails on it.

examples/ob09020 goes back to PUBLISHED VALUES with the justifying prose
removed -- a params file is a template, and a reader cannot tell a
transcribed number from a fitted one when every entry carries a defense.
pi_E_N returns to Skowron's -0.022 from a hand construction, q to the
printed 0.273, and theta_E = 2.95 is now pinned.  The set does not close,
which is the normal case for marginal medians from one table plus
physical parameters from another paper, and the new report says so
instead of the file hiding it behind tuned numbers.

bigomega is the one entry that cannot be "the published value": Yee's
Omega_node = -7.767 is in their frame, and the documented mapping
consumes phi_pi, which is a model output.  Re-converting against the
delivered phi_pi (162.068, from 158.795) gives 340.755, and the rule
reproduces the old 337.484 to 0.002 deg.  Start logp +19,024 -> +36,397,
with alpha_0 = 189.082 (published 189.08) and s_0 = 0.4264 (0.4294)
unchanged.  Measured: this fit cannot walk in bigomega at all -- +/-10
deg costs >1e6 nats and 60 deg costs 7e5, with the midpoint WORSE than
either end, so it is a dead mode and not a slow one.

The ob09020 acceptance fixture is regenerated for the deliberate seed
change (81 terms); the other 12 stay bit-identical.

Verified: 8 new unit tests (the units one fails on the pre-fix code);
85 passed across the acceptance, keplerian, polish, hot-chains and
wrapup-guard files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jdeast
jdeast force-pushed the worktree-user-start-check branch from ef7df4e to 11c4430 Compare September 11, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant