# Fix v4.33 fallout in the ignored kernel suites: Prop-below call-site surgery + egress hints channel - #578
Merged
Merged
Conversation
The IX_APP_DIFF AppTypeMismatch trace logged through the log facade, but no log backend is installed in the CLI or test binaries, so every dump was silently dropped — the same failure mode #571 fixed for the inductive.rs canonicity dumps. Emit on stderr like those, and include the failing constant's debug label in the header line since the dump is only legible next to the constant it fired in.
rebuild_named copied a Named entry's meta and original fields but dropped hints. Decompile reconstructs DefinitionVal.hints from the per-name channel (absent → Opaque), and roundtrip_block Phase B validates each decompiled aux constant's Lean-level hash — which includes hints — against the source env. On the kernel egress leg every regenerated .casesOn/.recOn/.below/.brecOn therefore failed that validation (Opaque vs Abbrev, 4846 spurious Pass-2 errors on a full env), recovery's own hash check failed the same way, and the raw generated form shipped in place of the metadata-restored one. That substitution was invisible while raw regeneration was name-faithful; it surfaced as kernel-ixon-roundtrip mismatches once two things landed on top of it: #571's Phase-3b below.casesOn regeneration (whose below-rec input is regenerated in the decompile work env, where the original below ctors are not yet present, so the IH binder falls back to a bare `ih`) and v4.33's _hygCtx hygienic binder names in IndPredBelow output (ih._@.<module>.<hash>._hygCtx ._hyg.N), which made the fallback visibly diverge for every inductive predicate's below.casesOn (Nat.le, List.Forall₂, DHashMap.Raw.WF, …). With hints preserved, Phase B's restoration lands and the full-env kernel roundtrip is clean again: 169194 constants verified, 0 errors, 0 aux_gen diffs.
On a Prop-valued MUTUAL inductive-predicate family, IndPredBelow's .below is itself a mutual inductive whose parameters are the parent params followed by the parent motives — in Lean's source member order. The canonical below family is generated from the canonical parent rec, so when sort_consts orders the members differently from source (the BelowPredicate EvenP/OddP fixture flipped to (OddP, EvenP) under the v4.33 content hashes), the canonical below's motive-parameter order diverges from Lean's authored one. .below HEAD applications in user constants were already reconciled by below_call_site_plans, and .brecOn by its derived plans — but a Prop-below family exposes two more user-referencable surfaces with the same leading params+motives telescope: the .below CONSTRUCTORS and the regenerated .below.casesOn wrapper. Matchers compiled by structural recursion over the predicates reference both (the BelowPredicate match_2 matchers apply OddP.below.succ / EvenP.below.zero and OddP.below.casesOn with source-order motives), so the kernel rejected them with AppTypeMismatch — 2 failures in kernel-check-env. Register the same derived plan under each below ctor name and the .below.casesOn name (Prop families only — Type-level .below is a definition with neither). The apply site discriminates the two telescope shapes by the key's last component (below/below_N = head, which keeps its indices+major application floor; anything else = family member, with NO floor, since a field-less below ctor like EvenP.below.zero is fully applied at exactly params+motives). In both shapes everything after the motive segment is kept in place, so the below branch now emits one identity tail — byte-identical entries to the old fixed/extra split for heads. X.below.rec is deliberately not registered: only regenerated wrappers reference it, and those skip surgery via the aux-regen guard. Mirrored in lockstep across the Rust and pure-Lean compilers (apply, compile-side registration, decompile-side plan install — the latter derives the ctor names from the parent inductive via the same suffix transplant buildBelowIndcCtor uses, since the below inductive is not yet regenerated at install time). CallSite metadata makes the rewrite self-describing, so decompile restores the source-order spelling with no changes; aux-gen-diff and decompile-diff gates stay byte-identical and validate-aux is clean.
focusConsts gains the two _private BelowPredicate match_2 matchers — the constants that pinned the Prop-below ctor/casesOn call-site gap — so kernel-check-const reproduces that class of regression in ~10s without a full-env pass. Tests/Main.lean gains a kernel-roundtrip-ns=<prefixes> special arg mirroring rust-compile: the same compile → ingress → egress → decompile → hash-compare pipeline as kernel-ixon-roundtrip, but over the transitive closure of the name-prefix-matched constants only. kernel-roundtrip-ns=Nat.le reproduced the below.casesOn binder-name regression on a 98-constant closure in 40ms.
johnchandlerburnham
force-pushed
the
jcb/fixup
branch
from
August 20, 2026 09:08
f840a2e to
6834b0c
Compare
One over-split method chain in install_decompile_call_site_plans from the below-family plan registration; whitespace-only.
Since #575 stopped exporting RUSTFLAGS, .cargo/config.toml's -Ctarget-cpu=native is authoritative in CI. That is the point for the warp jobs (bench artifact reuse, with CPU provenance logged), but rust-test runs on ubuntu-latest — a heterogeneous fleet sharing one cargo cache. A proc-macro dylib or test binary compiled native on an AVX-512 runner SIGILLs when a later run lands on a non-AVX-512 one; clippy-driver died with SIGILL checking aiur off a cached tracing-attributes exactly this way, and nextest --run-ignored all executes release test binaries with the same exposure. Pin the job to -Ctarget-cpu=x86-64-v3 (AVX2 — the baseline every GitHub-hosted x64 runner provides). RUSTFLAGS participates in the rust-cache key, so this also re-keys away from the poisoned native-built cache entries.
arthurpaulino
previously approved these changes
Aug 20, 2026
johnchandlerburnham
enabled auto-merge (squash)
August 20, 2026 09:56
The merge of main in c3f89ab kept rust-test on ubuntu-latest with the portable-codegen RUSTFLAGS pin from 05f6749, while also taking #577's setup-rust-toolchain, which now hard-requires a Warp Linux x64 runner — so the job failed the guard immediately. Both were fixes for the same native-codegen SIGILL flake, racing by twenty minutes; #577 resolved it by moving rust-test onto the Warp x64 fleet instead. Take main's side wholesale: the RUSTFLAGS pin is moot on the homogeneous Warp fleet, and keeping it there would override the authoritative .cargo/config.toml -Ctarget-cpu=native and re-key the warp-x64 cache with non-native binaries.
arthurpaulino
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two failures in the post-v4.33 (#572) ignored sweep:
Both are latent bugs that the toolchain bump exposed, not regressions in #572
itself. Neither suite runs in PR CI (
--ignored), so they surfaced only on alocal sweep.
Bug 1 — Prop-mutual
.belowfamilies: ctors andbelow.casesOnhad no call-site surgery (kernel-check-env)The two failing constants were the
_private.…BelowPredicate.{oddp,evenp}_nonneg.match_2matchers from #571's Prop-mutual fixture, rejected by the kernel with
AppTypeMismatch at depth=0.Mechanism. On a Prop-valued mutual inductive-predicate family,
IndPredBelow's
.belowis itself a mutual inductive whose parameters are theparent params followed by the parent motives — in Lean's source member
order. Our canonical below family is generated from the canonical parent
recursor, so its motive-parameter order follows
sort_consts' canonical classorder instead. The two orders coincided for
EvenP/OddPuntil the v4.33content hashes flipped the structural sort to
(OddP, EvenP):User references to
X.belowheads are already reconciled bybelow_call_site_plans, andX.brecOnby its derived plans. But a Prop-levelbelow family exposes two more user-referencable surfaces with the same leading
params, motivestelescope, and neither had a plan:OddP.below.succ,EvenP.below.zero, …), andX.below.casesOnwrapper.Matchers compiled by structural recursion over the predicates reference both
with source-order motives.
IX_APP_DIFF(see Bug 3) showed the kernel'scanonical
OddP.below.succreceiving the EvenP motive where it expects OddP's:Fix. Register the same derived
BRecOnCallSitePlanunder each below-ctorname and the
.below.casesOnname (Prop families only — Type-level.belowis a definition with neither), in the existing
below_call_site_plansmap.The apply site discriminates the two telescope shapes by the key's last
component (
below/below_N= head, which keeps its indices+major applicationfloor; anything else = family member, with no floor, since a field-less
below ctor like
EvenP.below.zerois fully applied at exactlyparams+motives). In both shapes everything after the motive segment is kept in
place, so the below branch now emits one identity tail — byte-identical
entries to the old fixed/extra split for heads.
X.below.recis deliberatelynot registered: only regenerated wrappers reference it, and those skip surgery
via the aux-regen guard.
Mirrored in lockstep across the Rust and pure-Lean compilers (apply branch,
compile-side registration, decompile-side plan install — the latter derives
the ctor names from the parent inductive via the same suffix transplant
buildBelowIndcCtoruses, since the below inductive is not yet regenerated atinstall time). CallSite metadata makes the rewrite self-describing, so the
decompiler restores the source-order spelling with no changes.
This keeps the canonicity architecture as documented (§9.3/§17.2: canonical
bytes are hash-sorted; the Lean-facing view is restored via surgery metadata),
rather than source-ordering the below generation — the alternative considered
and rejected because it would re-introduce source-order dependence into
canonical content.
Bug 2 —
ixon_egressdropped the per-namehintschannel (kernel-ixon-roundtrip)The reported "50 mismatches" is
compare_envs' error cap, not the total; allwere
X.below.casesOnof inductive predicates (Nat.le,List.Forall₂,Std.DHashMap.Raw.WF, …) whose IH binder decompiled as a bareihwherev4.33's IndPredBelow now writes hygienic
ih._@.<module>.<hash>._hygCtx._hyg.Nnames.
Mechanism.
kernel_egress.rs::rebuild_namedcopied aNamedentry'smetaandoriginalfields but droppedhints. Decompile reconstructsDefinitionVal.hintsfrom that per-name channel (absent →Opaque), androundtrip_blockPhase B validates each decompiled aux constant's Lean-levelhash — which includes hints — against the source env. On the kernel egress leg
every regenerated
.casesOn/.recOn/.below/.brecOntherefore failed thatvalidation (
OpaquevsAbbrev; 4,846 spurious Pass-2 errors on a full env),recovery's own hash check failed the same way, and the raw generated form
shipped in place of the metadata-restored one.
That substitution was invisible while raw regeneration was name-faithful. It
became visible once two things landed on top of it: #571's Phase-3b
below.casesOnregeneration (whose below-rec input is regenerated in thedecompile work env, where the original below ctors are not yet present, so the
IH binder name falls back to a bare
ih), and v4.33's_hygCtxhygienicbinder names, which made the fallback diverge from the original for every
inductive predicate's
below.casesOn.Fix. One line: carry
hintsthroughrebuild_named(plus the doccomment). Phase B's metadata restoration now lands, which also shields the
Lean-facing env from any future generation-side binder-name drift.
Bug 3 — the kernel's
[app diff]dump was silently droppedIX_APP_DIFF's AppTypeMismatch trace logged through thelogfacade, but nolog backend is installed in the CLI or test binaries — the same failure mode
#571's first patch fixed for the inductive.rs canonicity dumps. Emitted on
stderr now, with the failing constant's debug label in the header. (This dump
is how Bug 1 was diagnosed.)
Tests / tooling
focusConstsgains the two BelowPredicatematch_2matchers, sokernel-check-constreproduces this regression class in ~10s(
IX_KERNEL_FOCUS_CONST=match_2 lake test -- kernel-check-const --ignored).Tests/Main.leangains akernel-roundtrip-ns=<prefixes>special argmirroring
rust-compile: the same compile → ingress → egress → decompile →hash-compare pipeline as
kernel-ixon-roundtrip, but over the transitiveclosure of the name-prefix-matched constants only.
kernel-roundtrip-ns=Nat.lereproduced Bug 2 on a 98-constant closure in40 ms.
Verification
kernel-check-env: 219,554/219,554 passed (was 219,549/219,551)kernel-ixon-roundtrip: 169,194 constants verified, 0 errors (the oldrun stopped at the 50-error cap partway through the env); Pass-2
aux_gen differrors: 4,846 → 0validate-aux: 0 failures;aux-gen-diff: all gates PASS (Lean/Rustbyte-identical, incl. plans + parallel driver gates);
decompile-diff: allgates PASS incl. callsite-replay
lake test -- --ignored --exclude=tc-pins,tc-accel-diff,tc-anon-diff,tc-init,tc-tutorial,tc-roundtrip,lean4lean→ exit 0
Non-blocking observations
The
[compile_env] block FAILED inductBadNonSort*/inductTooFewParams/r6Host.mkstderr lines during the sweep are expected kernel-tutorial noise
(intentionally-bad fixtures being rejected). However,
r6Hostis nowrejected at compile time (
missing constant: r6Host.rec_1— the fixtureenv lacks the nested-aux recursor a real Lean env would carry) rather than
by the kernel positivity descent it was written to exercise. Follow-up: give
the fixture a dummy
rec_1so the R6 descent path is exercised again.X.below.recheads and partial applications of below-family constants belowthe params+motives floor remain outside surgery's reach, matching the
existing
below/brecOnhead behavior — such call sites fail loudly in thekernel check rather than silently miscompiling.
Also in this PR
style: rustfmt— one over-split chain in the new plan-registration code.ci: portable codegen for the GitHub-hosted rust-test job— the lint job'sclippy failure was not a lint: since ci:
!benchmarkfixes #575 made.cargo/config.toml's-Ctarget-cpu=nativeauthoritative in CI, the one job on the heterogeneousubuntu-latestfleet could execute native-built cached artifacts (e.g. atracing-attributesproc-macro dylib insideclippy-driver) on a runnerwithout the builder's ISA extensions → SIGILL;
nextest --run-ignored allhad the same runtime exposure. Pin that job to
-Ctarget-cpu=x86-64-v3(the GitHub-hosted AVX2 baseline) — RUSTFLAGS is in the rust-cache key, so
this also re-keys away from poisoned caches. Warp jobs keep
nativeperci:
!benchmarkfixes #575's bench design. Affects every PR's CI, so this hunk can becherry-picked to main independently if preferred.