HermitCrab optimization ledger: 22 attempts, no speedup, one located target - #490
Open
johnml1135 wants to merge 5 commits into
Open
HermitCrab optimization ledger: 22 attempts, no speedup, one located target#490johnml1135 wants to merge 5 commits into
johnml1135 wants to merge 5 commits into
Conversation
Single consolidated report from a measurement effort across 33 conformance grammars plus Sena, Indonesian and Amharic. One row per optimization attempted, with what was expected, why it failed, and the number that settled it. Net speed delivered: none. The fastest thing tried was 4% slower. What the work produced is the measurement, a closed family of optimizations, a located target, and this ledger. Headline findings: - Synthesis is ~5% of Sena and 0.3% of Amharic wall time. Four prior optimization rounds and the originating proposal were aimed at the wrong phase. - The redundancy in synthesis is apparent, not real: three independent measurements (9,774x, 6,476x, 3.22x/8.10x) all collapsed once the key carried the trail, the last to hits=0. That closes packed forests, fold sharing and synthesis-input dedupe as a family. - Only ~16% of the Amharic analysis cascade is linguistic computation; clone plus unattributed orchestration is 84%. Docs-only by design. All instrumentation stays on the four named branches: the probes gate on volatile bool reads inside Matcher.cs, a hot inner loop in SIL.Machine, which is fine for a measurement run and not something to carry in product code. Rebuilding a probe from a branch is cheap. Records two durable code constraints, a calibration record of predictions vs outcomes, and eight method rules -- including that a bucket reporting exactly 0.00 is broken until proven otherwise, and that an explicit remainder column would have caught four of the five retractions at the point of measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 27, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #490 +/- ##
=======================================
Coverage 73.33% 73.33%
=======================================
Files 445 445
Lines 37317 37317
Branches 5118 5118
=======================================
Hits 27367 27367
Misses 8825 8825
Partials 1125 1125 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1. '6 of 7 heavy words exactly 1.00' was FALSE -- the source table shows 4 of 7. The three that moved (1.13, 1.16, 1.38) all return zero parses, so R tracks failure not size, and cinacemerwa -- the corpus's most expensive word -- cleared the 1.3 gate. Row 4's flatness was overstated. Restores the reopening condition and the nogood observation (434,628 vs 25,102), both dropped from the source. 2. The Sena bucket row spliced two runs: battery 51.4% / cascade 18.4% from the 143,303ms run, ExpandAlternatives 20.3% / unaccounted 1.5% from the 112,713ms run. Together they sum past 100%, violating this document's own method rule 8. Now labelled per run and marked not co-summable. 3. 'The engine spends its time constructing and discarding objects, not deciding linguistic questions' overreached -- 37.4% is unattributed, i.e. unknown rather than known-non-linguistic -- and contradicted the fired gate three lines below. 4. Row 20 claimed generation's ratios 'would apply at face value', which is exactly the artifact rows 5/6 closed. Generation changes the share, not the soundness argument, which is workload-independent. Also: scopes the family claim to merge-by-key forests, since row 19's contexted-constraint variant packs WITHOUT merging and the argument does not reach it; softens the Amharic state-floor claim to inferred; downgrades row 14 to adjacent evidence; moves row 21 (CLOSED) into the closed table; and adds gates for the clone-discard fraction, the Sena battery decomposition (potentially ~20% of Sena wall, which would outrank every open row) and the generation probe. Row 16 is flagged as likely understated at 3-8%, since 85% of its duplicates are same-analysis-word and the same-word ceiling is nearer 17%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Consolidates rows 14, 16, 17, 18, 19, 20 and 22 from the six probe branches that measured them. Net result of the round: no optimization survived, and the ledger now stands at 21 closed / 1 partly open out of 22. Highlights: - Row 18 is worse than a no-op, not neutral: Sena 300 words runs 28-32% SLOWER under a shared scope with ~29.3 GB peak, because MaxMemoEntries is a per-SCOPE cap -- two heavy early words exhaust it and every later word then loses the within-word memoization a fresh scope would have given it. Cross-word hits are negative (-77,648 memo, -747,607 nogood). - Row 16's sound census: 395,026 alternatives -> 351,414 distinct (89.0%), ceiling 2.3-2.5%. Two of three Sena words have zero real duplicates. - Row 14: every nogood bucket is a singleton, 0.0% subsumable, across Sena and all 33 fixtures. - Row 19: shape-free die-point cost 0.0-6.0%, and total rejection cost is 0.2% of wall -- two independent reasons to close. - Row 22: Sena's battery does not understate the prefilter cap; 21.2% matcher share gives ~10.1%, the same bound as Amharic's. - Row 20: generation hits 2.67% pooled, 37.7% on its best fixture, and still loses to noise. - Row 17 remains partly open: the ~38% cascade remainder is unattributed, Equality and Memo and Trail each measured <0.2%, and GC pause time is the leading candidate but netstandard2.0 cannot measure it. Zero duplicate tokens on every corpus including full Sena (7,121) and Amharic (673), so type-level caching was never available either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…base The previous fold-in (bc80bf0) was built from feature/per-node-cost, which predates the corrections made in the prior review round. It silently reverted them and dropped ~93 lines (213 -> 144). Rebuilt as: corrected base f66b014 + the seven verified row cells + this round's additions. Restored, all of which bc80bf0 had deleted: the '4 of 7 heavy words' correction (bc80bf0 had regressed to the false '6 of 7'); the Sena two-run splice disclosure ('not co-summable'); the two durable code constraints, including the Word.ReplayOnto / _mrulesUnapplied hazard that the PR body advertises; the 'net speed delivered: none' section; the calibration record; and the 'merge-by-key' family qualifier. Added this round: method rules 6-9 as a union rather than one branch's subset (min-of-N vs spread; never dispatch timing probes concurrently; a shared bounded cache without eviction can be worse than no sharing; never aggregate the test logger's doubled output). Corrected two things in row 17(a). The ~38% remainder came from a CONTENDED run, and descheduled wall time lands preferentially in the remainder while deflating self-time shares -- CPU starvation is an unbracketed candidate biasing exactly that way. And 'netstandard2.0 cannot measure GC pause' is wrong at the measurement point: only the libraries are netstandard2.0, while the probe harness targets net10.0, so GC.GetTotalPauseDuration() is directly callable. Gated two-step probe recorded. Also fixed an internal contradiction (the clone caution still called the discard fraction unmeasured after row 17b measured it at 66.2%), added row 22's bucket-exclusivity caveat, retitled the Open section now that it holds outcomes, and extended the evidence table to all seven probe branches -- which are now pushed, since they were local-only and one worktree prune from gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…anches The seven probe branches are being deleted, which would leave the ledger's evidence table dangling. hermitcrab-probe-design.md replaces it. It carries: the two-assembly probe architecture and the gate trap that makes a bucket read exactly 0.00ms; the eight exclusive wall-time slices and why synForward must be net of cascade+battery (the 25x error); the per-node decomposition and how it avoids double-counting across assemblies; the census designs and why die points must be cost-weighted (100% of events, 0.2% of time); all three key designs with field-level audits, including why the ordered remaining trail is load-bearing and why Word.ValueEquals is not a starting point; the RuleLengthClassifier rules and the un-application direction trap; the full A/B methodology with the three failures that forced each part; and the shared-scope concurrency hazard. Restates the two durable engine facts standalone so they survive independently of any row: ReplayOnto not splicing _mrulesUnapplied, and MaxMemoEntries being a per-scope cap with no eviction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two documents, no code. HermitCrab has been optimised repeatedly without anyone measuring where its time goes; this measures it across 33 grammars plus three real corpora, and records 22 attempts so none of them is tried a third time.
Net speed delivered: none. Stated first because the value here is negative results. The fastest thing actually built ran 4% slower; a second ran 28–32% slower. Nothing in the closed section shipped. The existing wins in this codebase — the affix-template memo (30.5 s → 6.1 s), the mrule cascade memo, shape sharing — all predate this work.
Where the time actually is. Not synthesis. Synthesis is ~5% of Sena and 0.3% of Amharic. Four prior optimisation rounds and the proposal that started this were aimed at that phase. The cost is analysis, and which part differs by grammar: Sena is affix-template-battery-bound, Amharic is cascade-bound at ~170 ms per state in a cascade already at its state floor. Inside that cascade, only ~16% is identified as linguistic computation — matching plus feature unification. Clone/freeze is 46.7% and 37.4% is unattributed.
A family closed, not a candidate. Forward synthesis looked highly redundant four separate times, at four different boundaries. Each collapsed once the key carried the ordered remaining trail: 9,774x → 15–40%; 61-of-395,026 → 351,414-of-395,026; 3.22x/8.10x →
hits = 0. The trail is what makes each step distinct, so packed forests of the merge-by-key kind, fold-step sharing and synthesis-input dedupe all fail for one shared reason. Two of three Sena words have zero real duplicate alternatives. Measured on parsing; generation was measured separately and behaves the same.One structural finding beyond "doesn't pay": corpus-scope memoisation is actively harmful, not neutral —
AnalysisScope.MaxMemoEntriesis a per-scope cap with no eviction, so two heavy Sena words exhaust it and every later word loses the memoisation a fresh scope would have given it. Cross-word hits go negative.What to review.
docs/hermitcrab-optimization-ledger.mdis the register — 22 rows, tried / closed / open, each with the number that settled it.docs/hermitcrab-probe-design.mdis the design record: enough to rebuild any probe in a day, since the seven probe branches were deleted rather than maintained (they gate onvolatile boolreads insideMatcher.cs, a hot inner loop inSIL.Machine).Standing: 21 closed, 1 partly open. The only live thread is the ~38% cascade remainder — and its measurement was taken under CPU contention, so starvation is itself an unbracketed candidate. A gated two-step probe is recorded.
One code change is worth making on its own merits, deliberately not in this PR:
Word.ReplayOntodoes not splice_mrulesUnapplied, and is safe only becauseAnalysisStateKeyincludes the count multiset. Anyone narrowing that key breaks correctness with no test failure. ~10 lines plus a test that is a no-op under today's key.Category results — every group at the 2x bar
Ratios are deterministic and reproduce byte-identically across runs. Only
wallMs >= 50rows can carry a timing claim: one 10 ms fixture's share moved 1.3% to 14.5% between two runs of identical code.The categories are real and the 2x was real as available sharing — the fixture built to be order-dependent reports exactly 1.00x, which is the mechanism failing where it should. It is not extractable, because what makes it visible is what makes it unsound. F through K are retained rather than deleted: a category that fails on today's fixtures may pass on a grammar we do not yet have.
The seven rows reopened and measured in the final round
ExpandAlternativesdedupeZero duplicate tokens on every corpus, including full Sena (7,121 words) and Amharic (673), so type-level caching was never available either.
Corrections made during this work
Five published findings were withdrawn on measurement. Four were arithmetic on an unmeasured denominator; the fifth was a contended timing.
synForward, which excludes the cascade and battery buckets the shared work runs inside. The largest fixture's real share is 73.9%, the highest of any reliable row — understating Sena's ceiling 25x.cinacemerwa— the most expensive word in the corpus — cleared the gate at 1.38x.A sixth was caught before publication: a first A/B read 1.53x that was entirely JIT warm-up. The methodology in the design doc — discard warm-up, min of N interleaved, print the off-arm spread beside every speedup — exists because of these.
This change is