diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md new file mode 100644 index 00000000..f4c8228d --- /dev/null +++ b/docs/hermitcrab-optimization-ledger.md @@ -0,0 +1,229 @@ +# HermitCrab optimization ledger — tried, closed, do not retry + +**Purpose: stop this work being redone.** One row per optimization attempted, what was expected, +what actually happened, and the number that settled it. If you are about to try something on this +list, read its row first — several of these look irresistible on paper and three of them have +already been independently rediscovered and re-retracted. + +Deliberately in-repo and durable, not evicted into a PR accordion. A closed avenue is only closed +if the next person can find out cheaply that it is closed. + +**This is the only report.** The working docs it was distilled from +(`hermitcrab-packed-forest-research.md`, `hermitcrab-forest-memo-plan.md`, +`hermitcrab-forest-memo-ceiling.md`, `hermitcrab-synthesis-fold-probes.md`) are chronological +process records containing retracted intermediate claims, and are deliberately **not** merged. +They are preserved only as this record. **The evidence branches were deleted after the round +closed**, along with all instrumentation: research scaffolding is not worth maintaining in two +product assemblies, since the probes gate on `volatile bool` reads inside `Matcher.cs`, a hot inner +loop in `SIL.Machine` used by every consumer. + +**`docs/hermitcrab-probe-design.md` is the replacement.** It carries the probe architecture and its +two-assembly gate trap, the eight exclusive wall-time buckets and why `synForward` must be recorded +net of cascade and battery, the per-node decomposition, the census designs, all three key designs +with field-level audits, the classifier's direction trap, and the full A/B methodology with the +three measurement failures that forced each part of it — enough to rebuild any probe in a day. The +expensive part was never the code; it was learning what to measure and which measurements lie. + +**No instrumentation is merged, by design.** The probes gate on `volatile bool` reads inside +`Matcher.cs` — a hot inner loop in `SIL.Machine` used by every consumer — which is an acceptable +cost for a measurement run and not one to carry in product code indefinitely. Rebuilding a probe +from a branch is cheap; maintaining research scaffolding in two product assemblies is not. + +## Net speed delivered: none + +No optimization in the "closed" section shipped, and none of them made anything faster. The +fastest thing tried was 4% **slower**. The pre-existing wins in this codebase (rows 1–3) predate +this work. What this effort produced is the measurement, the closed family, the located target, +and this ledger. + +--- + +## Shipped and working + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 1 | Memoize analysis morphological-rule cascade | Same state reached by many rule orders; cache it | Works, but the cascade was never the cost | 2,555 expansions vs a 2,546-state floor; 1.4 s of a 30.5 s word | +| 2 | Memoize affix-template battery (same key) | Battery was 93% of wall time | **The big win.** Reduced how *often* the battery runs | 38,840 runs -> 2,581; word 30.5 s -> **6.1 s** | +| 3 | Shape sharing at clone (`CloneShareFrozenShape`) | Eager deep-copy of Shape dominates allocation | Works, allocation only | **-4.5 to -7.6% bytes**, no wall change | + +## Closed — measured, does not pay + +| # | Optimization | Expected | Why it failed | Number | +| --- | --- | --- | --- | --- | +| 4 | **Narrow `AnalysisStateKey`** (drop strictly-shrinking rules' un-application counts) | `MaxApplicationCount` defaults to 1, so the count multiset is "which subset of rules was used" — 2^27 states. Dropping 19 of 27 rules gives 2^8 | Shape + syntactic FS + realizational FS *already* discriminate. The count component was implied by its neighbours. A worst-case bound collapsing is not the realised state count collapsing | Realised state collapse **Sena 1.12x** (state-weighted) vs a 1.3 gate; **4 of 7** heavy words exactly 1.00, `atawirambo` 2,556 -> 2,556. The three that moved -- `pidafikawo` 1.13, `cinagumanika` 1.16, **`cinacemerwa` 1.38** -- all return **zero parses**: R tracks failure, not size, and the corpus's single most expensive word *did* clear the gate. **Reopening condition:** a grammar with genuinely high R; the census is cheap and is the right first question to ask of any new grammar | +| 5 | **Fold-step sharing memo** (share deterministic synthesis fold steps by computed value) | Probe measured 3.22x and 8.10x shareable steps; ~50% wall ceiling on two reliable fixtures | A sound key must carry the **ordered remaining trail**. Two candidates then have to agree on their entire future to share a step, which almost never happens | **hits = 0** on the best fixture; **0.96x** (4.5% *slower* — pure key-construction cost); 1.06x on the second, inside 21.9% noise | +| 6 | **Order-insensitive synthesis-input dedupe** | 9,774x duplicate synthesis inputs | HermitCrab's morphological rules are **not order-invariant**: same rule multiset, different order, different output. Merging loses parses | 2 violations Sena, 12 independently Indonesian (e.g. `{meN, -Cont}`). Sound version: **15–40%** | +| 7 | **Trail-position indexing of synthesis rules** | 11,445,538 of 11,445,538 rejections are "this rule is not the pending trail rule" — 100% of the histogram | Each rejection is one array index plus one reference compare. Count is not cost | 11.4M x 29 ns = 0.33 s of 143 s = **0.2%** | +| 8 | Synthesis-side length bound (Gate A) | Reject candidates that cannot reach the surface length | At the comparison point the candidate is still the bare root; its affix trail applies later inside `_synthesisRule.Apply`. Rejected valid parses | Unit suite 64 -> 34 passing. Reverted | +| 9 | Analysis length ceiling (Gate B) | Same idea, analysis side | Sound and correct. No available corpus contains the pathology it prunes | Byte-identical output, **no measurable speedup** (~4–7% slower, within noise) | +| 10 | Lexical reachability gate (Phase 5) | Prune candidates from which no lexicon root is reachable | Both reference corpora have real compounding in their deepest stratum, which disqualifies the gate everywhere | Proven **no-op** on both corpora | +| 11 | Tandem lexical intersection (T2) | Kill doomed branches early via the lexicon | The oracle only sees lexical unreachability. The expensive words fail on checks that run *after* lexical lookup succeeds | Pooled **23.5%** dead steps vs a 30% gate; one failure word at exactly **0.0%** dead | +| 21 | **Memo-key caching on the frozen `Word`** | **CLOSED.** Keys are built ~28x per distinct state, which looked like a cheap win. Measured cost of all `AnalysisStateKey` construction and hashing on Amharic: **3 ms of 81,707 ms = 0.0%.** Same error shape as #7: high count, no cost | +| 12 | Pool small short-lived collections | Fewer allocations, faster | `HashSet/Dictionary.Clear()` is O(capacity); Gen0 already beats pooling at this size | -15–17% bytes but **+8.6% wall**, +12% on the parallel battery. Reverted | + +## Closed before building — motivation refuted by measurement + +| # | Optimization | Expected | Why it died | +| --- | --- | --- | --- | +| 13 | Surface-length pruning inside the synthesis fold (P2) | "The only lever on Amharic's ~160 ms per synthesis run" | Amharic's forward synthesis is **14 ms**. The 160 ms was arithmetic on an unmeasured denominator — dividing 30 s by 186 synthesis inputs and assuming the time was there | +| 14 | Nogood lattice subsumption (P4) | Nogood hits dominate positive replays **434,628 to 25,102** — the engine mostly proves subtrees *empty*, and generalising "empty" over the feature lattice would turn nogoods from points into regions | **CLOSED on direct evidence** (it was previously closed on adjacent evidence, which is why it was reopened). Census over every empty-expansion key: `cinacemerwa` 16,176 expansions -> 16,176 distinct keys -> **16,176 singleton buckets, 0 subsumable (0.0%)**; `kukucitirani` 12,595 -> 12,595 -> **12,595 singletons, 0 subsumable**. Not one pair of nogood entries even shares a `(shape, stratum, counts, non-head)` bucket, so there is nothing for a subsumption ordering to act on. Breadth check across all 33 conformance fixtures agrees exactly: 7,493 empty expansions, 7,398 distinct keys, **0 subsumable (0.0%)** pooled, every non-empty bucket a singleton. Direction checked explicitly as `earlier.Subsumes(later)` (E subsumes Q, general over specific); the reverse would be unsound. Independently reconfirms row 4 from a different angle: the keys are maximally discriminating | +| 15 | Move constraints into the analysis key (Maxwell & Kaplan's category-splitting) | Their biggest measured win: make the chart prune what the constraint solver otherwise would | Everything synthesis rejects on is root- or realization-dependent, and the root is unknown until `LexicalLookup`. Also a Sena analysis state costs ~0.73 ms while a synthesis input costs at most ~0.12 ms — **states are dearer than synthesis inputs** | + +## Reopened this round, and their outcomes + +| # | Optimization | Status | +| --- | --- | --- | +| 22 | **Analysis-side rule prefilter** | 94.4% of Sena allomorph pattern matches fail (6,609,627 of 7,005,273) and 86.7% of 4,327,182 rule attempts reach matching, so the target is real | **CLOSED, cap ~10%.** The reopening hypothesis was that Sena's battery would badly understate the Amharic-derived ~11.5% bound. **It does not.** Sena `anBattery` matcher share is **21.2%**, against a >=30% gate — does not fire — giving a cap of 47.6% x 21.2% = **~10.1%**, essentially the same bound. Contended run; the gap to the gate is wide enough that contention will not close it. Caveat: the buckets are exclusive, so ~10.1% counts matcher *self*-time only — a prefilter would also avoid the failed attempts' allocations. Partially offset, though: the 21.2% includes successful matches a prefilter cannot avoid, and `AnalysisAffixTemplateRule.Apply`'s eager clone happens once per template *upstream* of any per-rule prefilter, so the prefilter cannot save it | +battery, which is only ~8% of its wall. **Sena's battery is 51.4% of its wall and has never been +decomposed per-node.** If its match share is comparable, the Sena cap is ~0.514 x 0.39 = **~20% of +wall**, outranking every other open row. Probe: `NodeCostProbe` over the 3 Sena heavy words; gate -- +matcher >=30% of Sena `anBattery` makes this the top open lead | +| 16 | **`ExpandAlternatives` dedupe** | 20.7% of Sena wall, the largest bucket nobody had instrumented. Apparent census: 395,026 alternatives -> **61** distinct (0.0%), 85% of duplicates same-analysis-word | **CLOSED on the sound census.** With the pending-trail content added to the key, 395,026 -> **351,414 distinct (89.0%)** -- only **11%** are duplicates at all, and all 43,612 of them are same-analysis-word. Sound ceiling = 20.7% x 0.110 = **2.28% of Sena wall**, against a gate of >=10% to build and <5% to close. **The fourth apparent redundancy to evaporate under a complete key** | +| 17 | **Per-node cost in the analysis cascade** | 84% of the Amharic cascade is not linguistic computation; clone is 46.7% and 37.4% was unattributed | **Partly settled, remainder still open.** (a) The remainder split **MISSES its <15% gate by ~2.5x** — still ~38% on Amharic `anCascade`. Memo, Trail and **Equality each measured <0.2%**, so `Word.ValueEquals` behind the cascade's `HashSet` is *not* the missing mass, contrary to the standing hypothesis. **Two caveats the first write-up missed.** (i) The ~38% figure comes from a *contended* run — three timing probes were live — and descheduled wall time lands preferentially in the remainder while deflating every self-time share, so **CPU starvation is itself an unbracketed candidate biasing in exactly the direction observed**. (ii) The claim that GC pause time is unmeasurable is **wrong at the measurement point**: only the library assemblies are netstandard2.0; the probe harness lives in `tests/SIL.Machine.Morphology.HermitCrab.Tests`, which targets **net10.0**, so `GC.GetTotalPauseDuration()` and `GC.GetGCMemoryInfo().PauseTimePercentage` are directly callable. Next probe, gated: re-run serial — remainder <20% means contention was the missing mass and 17(a) recloses on the cheaper-clone target; still >=30% then bracket with `GetTotalPauseDuration()` deltas — ΔPause >=50% of the remainder's absolute ms confirms GC (build target converges with clone/freeze at 46.7%), <=15% refutes it and the next split is enumerator/LINQ orchestration (the 2026-07 alloc audit already flagged `ExecuteCommands` enumerator boxing); a collection census over 24 Amharic words gives gen0 8,217 / gen1 2,729 / gen2 301, an unusually high gen1+gen2 ratio consistent with — not proof of — GC as the remainder. (b) **Clone-discard fraction = 66.2%** (created 2,505, discarded 1,659), a count and therefore contention-immune, against a >=70% gate: **misses by ~4 points, so the build is "make a clone cheaper", not "defer cloning"**. Scoped to `AnalysisAffixTemplateRule`'s eager per-template clone, the one clone-before-verification site; the other four analysis-side sites clone only after a successful match and were excluded rather than blended in | +| 18 | Corpus-scope memoization | `AnalysisScope` dies per word, but `AnalysisStateKey` is word-independent by construction, so entries are reusable across a corpus. Nogood hits dominate positive replays 10-17x, making cross-word nogood reuse the cheapest available form of reuse | **CLOSED, and worse than a no-op.** Indonesian, 121 words, **serial on a clean machine: 1.015x, 1.4% wall-clock reduction** against a >=20% build gate and <5% close gate. Costs **68 MB** added peak managed memory. Cross-word reuse is real but small: 39 memo, 73 nogood, 237 template-nogood hits. Token-vs-type: **121 tokens, 121 distinct**, so no type-level dedupe is available either. **A first, contended run of the same harness reported 2.385x / 58.1%** with a 210.9% spread on the shared arm against 13.7% on the baseline; two testhosts were live. The counts were byte-identical across both runs — only the timing was contaminated, and min-of-N on a high-spread arm picks the luckiest sample. Analysis-set equality held throughout, so the key genuinely is word-independent. **Sena, first 300 words, is the decisive case: the shared scope is 28-32% *slower* than per-word scopes** (two clean serial runs, no contention: 282.7s vs 372.2s and 308.9s vs 396.7s baseline-vs-shared; deterministic counters byte-identical across both runs), and costs **~29.3 GB** added peak managed memory against a 2 GB budget. Root cause: `AnalysisScope.MaxMemoEntries` (100,000) is a *per-scope*, not per-word, budget. Two heavy early words (#74, #76 of 300) alone fill the shared `Memo` table to the cap and `TemplateMemo` to 55,071; every word after that loses the free within-word memoization a fresh per-word scope would have given it for nothing, because `Store` silently no-ops past the cap (see its comment) with no eviction. Cross-word hits (shared minus baseline) are **negative** — memo -77,648, nogood -747,607 — i.e. sharing produced *fewer* total hits than 300 independent fresh-table runs would have on their own; the small `+820` template-hit / `+9` template-nogood gain does not come close to covering it. Amharic, capped at 28 words (word 29 hangs, excluded per standing guidance): **0.9% wall-clock reduction**, table nowhere near the cap (Memo 1,251, TemplateMemo 652 of 100,000), consistent with row 17's finding that Amharic's cost is dominated by clone/allocation per node rather than distinct-state count, so avoiding a handful of cross-word re-derivations barely moves wall time. Token-vs-type on all three corpora, at the runs' own sizes and on each corpus's full word list: **zero duplicate tokens** — whole-word caching is not available as a cheaper alternative anywhere here. No analysis-set divergence on any corpus, at any scope regime: the stop condition that would have put rows 1-3 in question did not fire. **A shared-but-capped memo table is not "the same optimization, smaller ceiling" — past the cap it actively takes away the free per-word memoization the product already has, and retains every live entry for the run's full duration instead of per-word, which is where the 29 GB comes from.** Any future attempt needs real eviction (LRU, generational, or per-word sub-budgets) before this is worth re-measuring, and even then the >=20% gate looks unreachable given Indonesian's ceiling under an uncapped-in-practice regime was already only 1.4% | +| 19 | Contexted constraints / abstract feature-only replay | **CLOSED.** Cost-weighted the P1b die-point histogram (count → wall-time per die point, gated before the run: shape-free cost ≥60% builds, <30% closes). Sena `atawirambo` **6.0%**, `cinacemerwa` **0.0%**, Amharic 28 words **0.4%** — all fire the close gate, none come close to the middle band. All 33 conformance fixtures measured too (breadth, not part of the gate): 24 close, 3 fire the build gate (`morphotactic-attribute-breadth` 67.1%, `mpr-group-overwrite-without-realizational` 91.5%, `prefixal-discontinuous-slot-dependency` 71.8%), 3 land in the undecided 30–60% band, 2 have zero rejections, 1 has zero words. **All three build-gate fixtures are sub-50 ms** (38.15 ms / 1.49 ms / 4.28 ms wall) — the method-rules-earned "cannot carry a timing claim" flag, so they do not override the real-corpus verdict. On the three real grammars, count and cost agree in direction (`RuleNotApplicableOrPatternMismatch`, shape-dependent, dominates both). Among the fixtures, one reliable one (wall ≥50 ms) shows the count/cost divergence the gate exists to catch: `suffixing-extension-slot-ordering` (53 words, 63.98 ms) shape-free count 12.9% vs. cost 53.3% — cost roughly 4x count, same direction as the ApplicationCount/MprFeatures/FeatureUnification checks costing more per event than pattern-mismatch's near-O(1) trail check. (`feature-gating-breadth`, 28.6% vs. 56.8%, shows the same direction but is sub-50 ms and unreliable, same flag as the three build-gate fixtures.) Orthogonal to classification: die-point *rejection* cost itself is negligible against wall time on all three real-corpus runs regardless of class (atawirambo ~19.8 ms of 11,547 ms wall = 0.17%; cinacemerwa ~112.9 ms of 57,275 ms = 0.20%; Amharic ~88.8 ms of 91,456 ms = 0.10%) — reconfirming row 7/6.2's finding that rejections are cheap. Abstract replay would be optimizing a phase that costs approximately nothing on the two grammars this project treats as ground truth. Not built | +| 20 | **Row 5's family on `GenerateWords`** (generation, ~100% synthesis share vs parsing's ~5%/~0.3%) | Row 20 was unmeasured; generation's share is ~100% so row 5's fold ratios would apply at face value if they transferred. `UseSynthesisFoldMemo` had never been wired into `GenerateWords` (only into the parse-then-synthesize path) — wired in under the same untraced/`MaxDegreeOfParallelism==1` restriction to make this measurable at all | Hits are no longer uniformly zero: 3 of 23 lookup-reaching fixtures clear the 5% gate (diacritic-segments 33.3%, suffixing-vowel-harmony 28.6%, suffixing-evidential-adjacency-chain 37.7%; a 4th sits just under it at 4.8%), reopening row 5 narrowly for generation. But the reopened case recloses immediately: the A/B on the best fixture shows the wall-clock "win" sitting inside its own noise floor, and the pooled hit rate is dominated by `deep-optional-affix-nesting` (925 of 1,613 lookups) at hits=0, reproducing row 5's exact 4.5%-slower key-construction cost on the identical fixture | Parity: 0 divergences, 1,236 generations, 32/33 fixtures (33rd throws identically both sides). Best case: 130 hits / 5 reps (37.7% hit rate), **7.8%** wall-clock reduction inside a **22.2%** off-arm spread — not a result. 19/23 lookup-reaching fixtures at exactly **0%**; pooled hit rate **2.67%** (43/1,613), skewed by the one large zero-hit fixture | + +--- + +## The lesson that generalises + +Three independent measurements, three boundaries, same collapse: + +| measurement | key used | apparent | sound | +| --- | --- | --- | --- | +| synthesis-input dedupe (#6) | order-insensitive | 9,774x | 15–40% | +| fold-entry census | trail position only | 6,476x | not established | +| fold-step sharing (#5) | trail position only | 3.22x / 8.10x | **hits = 0** | + +**The redundancy in HermitCrab's synthesis is apparent, not real. The trail is what makes each +step distinct, and every measurement showing large shareable work is measuring a key that omits +it.** The thing that makes sharing visible is the thing that makes sharing wrong. + +That closes a *family*: packed parse forests **of the merge-by-key kind**, fold-step sharing and +synthesis-input dedupe all require distinct derivations to converge on a genuinely identical +state. In this engine they do not converge -- the same fact as the rules being non-order-invariant, +seen from the other side. **It does not close row 19**: a contexted-constraint forest packs +*without* merging states, so this argument does not reach it. Row 19 stays open and unmeasured. + +## Where the time actually is + +Measured with eight exclusive buckets summing to wall. Amharic is a single run (`unaccounted` +0.1%). **Sena is spliced from two runs -- see the note in its row.** + +| grammar | breakdown | +| --- | --- | +| **Sena** heavy words | battery **51.4%**, cascade 18.4%, all synthesis ~5% -- *pre-`ExpandAlternatives` run (143,303 ms, unaccounted 20.1%)*; `ExpandAlternatives` **20.3%**, unaccounted 1.5% -- *later run (112,713 ms)*. **Two runs: not co-summable.** One re-run would put all Sena shares on one denominator | +| **Amharic** | analysis cascade **~95%** of analysis, which is 99.5% of wall; **all synthesis 0.3%** | + +**Every optimization in this ledger that worked reduced how many nodes are visited. None touched +what a node costs.** Amharic spends **~170 ms per analysis state**. (The *measured* state-count floor -- 2,555 +expansions against 2,546 states -- is Sena `atawirambo`. For Amharic, near-floor behaviour is +**inferred** from R ~ 1, which compares the full key against the narrowed key, not against a +semantic floor.) + +### Inside one node (Amharic, 28 words, both probe layers live) + +| bucket | `anCascade` 81,707 ms | `anBattery` 7,368 ms | +| --- | --- | --- | +| clone / freeze / allocation | **46.7%** | 41.9% | +| **remainder (unattributed)** | **37.4%** | 15.4% | +| matcher traversal | 12.5% | **39.3%** | +| FeatureStruct algebra | 3.4% | 3.5% | +| memo-key construction | 0.0% | 0.0% | + +Censuses: 27,385 rule attempts — guards reject 19.5% (application-count 6.3%, unifiability 13.2%), +**80.5% reach matching**. Of 25,178 allomorph pattern matches, **1,938 succeed (7.7%)**, 23,240 +fail (**92.3%**). + +**The headline: only ~16% of the Amharic analysis cascade is *identified* as linguistic +computation** -- matching 12.5% plus feature unification 3.4%. Clone/freeze is **46.7%** and +**37.4% is unattributed**, which is unknown rather than known-non-linguistic. The defensible +statement is that at most a sixth of the cascade has been shown to decide linguistic questions; +of the rest, half is measured (clone) and half is still open. + +**Gate outcomes, written before the run.** "Any bucket >=50% becomes the sole build target" — clone +is 46.7%, does not fire. "Pattern matching dominates *and* >=80% of attempts fail -> build a +prefilter" — 92.3% do fail, but matching is 12.5%, so it does not dominate; does not fire. +**"Remainder >=30% means the decomposition is incomplete and that is the result" — 37.4%, FIRES.** + +So there is no declared build target yet. The next probe splits the 37.4%: `RuleCascade` / +enumerator orchestration, trail dictionary operations, LINQ allocation, memo lookups, GC. + +**A caution on the 46.7% before anyone acts on it.** `Word.Clone` being half the cascade does not +by itself mean cloning is waste — the cascade forks a `Word` per rule application precisely so it +can explore alternatives without mutating shared state. The deciding question is what fraction of +clones are *discarded unused*: cloned, rejected by a guard or a failed match, thrown away. Given +92.3% of pattern matches fail, that fraction has now been **measured at 66.2%** (row 17b), which misses the >=70% gate and points at the cheaper-clone build rather than the defer-clone one. If most +clones are discarded, the lever is **not cloning until a rule commits**. If most survive, the lever +is **making a clone cheaper** (copy-on-write shape, which the RUSTIFY branch already explored). +Different builds; measure first. **Gate: probe the clone-discard fraction; >=70% discarded means +the defer-clone build, below that the cheaper-clone build.** + +## Two durable code constraints + +Neither is expressible in code, and both bite silently. + +1. **`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and + `_nonHeadApps` only. This is safe *solely* because `AnalysisStateKey` includes the per-rule + count multiset, which guarantees a memo hit has identical arrival and stored-arrival counts. + **Anyone narrowing that key breaks correctness with no test failure.** Fix, if the key ever + changes: store arrival counts on `MemoEntry` and compute `stored - storedArrival + query` — a + no-op under today's key, which is also how to test it. This is the one code change from this + work worth making on its own merits, as defensive hardening. +2. **HermitCrab's morphological rules are not order-invariant.** Same pending-rule multiset, + different application order, different synthesis output — 2 cases on Sena, 12 independently on + Indonesian (e.g. `{meN, -Cont}`). This bounds every packed-readout scheme. It is the assumption + Maxwell and Kaplan (CL 19(4):571–590) rely on and this engine does not satisfy. + +## Calibration record — predictions vs outcomes + +Written before the measurements, scored after. Kept because the misses are informative. + +| prediction | outcome | +| --- | --- | +| Sena key collapse 2–4x | **1.12x** — wrong, and it was the number that decided the project | +| Indonesian key collapse 1.0–1.3x | 1.17x — right | +| Amharic "most likely to disappoint" | best static classification of the three (31 of 36 rules) — wrong | +| Fold sharing ~50% wall ceiling | **0.96x realised** — wrong; the ceiling was measured with a key that would lose parses | +| Amharic synthesis-bound at ~160 ms/run | forward synthesis is **14 ms** — wrong; arithmetic on an unmeasured denominator | + +## Method rules earned here + +1. **Counting is not timing.** Three published findings in this work were retracted, all arithmetic + on an unmeasured denominator: "synthesis is the bottleneck" (candidate counts), "Amharic is + synthesis-bound" (30 s / 186 inputs), "fixtures only look synthesis-heavy because they are small" + (divided by `synForward`, which excludes the buckets the work runs in). +2. **A ratio measured with an incomplete key is not a ratio.** See the table above, three times. +3. **Warm-up dominates small fixtures.** A first A/B here showed 1.53x purely from JIT; the off-arm + alone varied 42.7% between its own first and second sample. Discard warm-up, take min of N, + interleave arms, and print the off-arm spread as a noise floor beside every speedup. +4. **Sub-50 ms fixtures cannot carry a timing claim.** One 10 ms fixture's share moved 1.3% -> 14.5% + between two runs of identical code. +5. **Three grammars minimum, reported unpooled.** Maxwell & Kaplan measured a 100x swing between two + variants of one grammar. A pooled average across fixtures of different size has had to be + retracted twice in this project. +6. **Min-of-N is only valid when the arm's spread is small relative to the effect.** Row 18's + first run read 2.385x with a 210.9% spread on one arm and 13.7% on the other; serial and clean + it is **1.015x**. Min-of-N picks each arm's luckiest sample independently, so a high-spread arm + wins spuriously. Print the spread beside every speedup and refuse a number whose spread exceeds + it. +7. **Never dispatch timing-sensitive probes concurrently.** Three at once put four testhosts on the + machine and invalidated the bucket percentages in rows 17 and 22 as well as row 18's headline. + Counts survive contention; shares and speedups do not. +8. **A shared bounded cache without eviction is not a smaller version of the optimization** — it + competes with itself for a fixed budget and can land strictly worse than no sharing. Row 18's + shared scope produced *negative* cross-word hits. +9. **Never aggregate the test logger's console output.** It prints every line twice, the second + copy space-prefixed; a `^\s*` grep doubles every total. This produced a 2x error in row 20's + first write-up. +10. **Search completeness is never traded for speed.** HermitCrab is the permanent fallback engine + behind the FST work. +7. **A bucket reporting exactly `0.00` is broken until proven otherwise.** `NodeCostProbe.Enabled` + was declared and read in `Matcher.cs` but set by nothing — two gates in two assemblies, one + wired. Both lower-layer buckets read `0.00ms`, which looks like "measured and negligible" + rather than "never ran", and their time silently inflated the remainder. Wire cross-assembly + probe gates from a single place. +8. **Always carry an explicit remainder column.** It caught the wall-split misattribution and the + dead probe gate. Four of the five retractions in this work would have been caught at the point + of measurement by a remainder that did not sum. diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md new file mode 100644 index 00000000..b1bb04ea --- /dev/null +++ b/docs/hermitcrab-probe-design.md @@ -0,0 +1,226 @@ +# HermitCrab probe design — enough to rebuild it + +Companion to `hermitcrab-optimization-ledger.md`. That document records *what was tried and why it +failed*. This one records *how it was measured*, in enough detail to reconstruct the instrumentation +and the keys from scratch. + +**Why this exists:** the seven probe branches that produced the ledger's numbers were deleted after +the round closed. Research scaffolding in two product assemblies is not worth maintaining — the +probes gate on `volatile bool` reads inside `Matcher.cs`, a hot inner loop in `SIL.Machine` used by +every consumer. So the code is gone and this is the design record. Anything here can be rebuilt in a +day; the expensive part was learning what to measure and which measurements lie. + +--- + +## 1. Instrumentation architecture + +Two probe classes, in two assemblies, with **two independent gates**. + +`SynthesisProbe` in `SIL.Machine.Morphology.HermitCrab`, and `NodeCostProbe` in `SIL.Machine`. Each +has its own `internal static volatile bool Enabled`, default `false`. **Both must be set.** Setting +only the HermitCrab one makes every lower-layer bucket read exactly `0.00ms`, which looks like +"measured and negligible" rather than "never ran" — this cost a round (ledger method rule: a bucket +reporting exactly zero is broken until proven otherwise). Wire both from one place in the harness. + +Everything is insertion-only: a `RecordDie(...)` or `Add*Ticks(...)` call placed immediately before +an existing `return` / `continue`, never a change to control flow. Where a site read +`else if (IsTracing) { trace }`, it becomes `else { if (IsTracing) { trace } Record(...) }` — +behaviour-preserving. Counters are `Interlocked`; timers are `Stopwatch.GetTimestamp()` deltas +accumulated with `Interlocked.Add`, with `Reset*` methods so a harness reads per-word deltas. + +### The wall-time split + +**Eight mutually exclusive top-level slices that sum to wall**, measured around +`Morpher.ParseWord`: + +`lookup` (brackets `LexicalLookup`, materialised with `.ToList()` so the bracket covers only the +lookup itself) · `synCascade` · `synBattery` · `synForward` · `synExpand` (brackets +`Word.ExpandAlternatives()` in `SynthesizeSequential`) · `anTotal` (brackets the whole +`_analysisRule.Apply(input)` call) · `unaccounted` (defined as the remainder, never tracked). + +`synForward` is recorded **net of** `synCascade` and `synBattery`: capture both counters before the +`_synthesisRule.Apply` call and subtract their deltas after. This matters enormously and caused a +25x published error — the work a synthesis fold memo would share runs *inside* the cascade and +battery brackets, so the share to divide by is `synCascade + synBattery + synForward`, not +`synForward` alone. Template slot rules compile to the same `SynthesisAffixProcessRule` classes via +`RuleBatch` (`SynthesisAffixTemplateRule` ctor), which is why they land in `synBattery`. + +Nested *inside* `anTotal`, as a breakdown rather than additional slices: `anCascade` (the +`_mrulesRule.Apply` call in `AnalysisStratumRule.ApplyMorphologicalRules`), `anBattery` +(`ApplyTemplateBattery`), `anPhono` (`_prulesRule.Apply`). + +**Always print the remainder as its own column.** It caught the misattributed split, the dead probe +gate, and would have caught four of five retractions at the point of measurement. + +### The per-node decomposition + +Inside `anCascade` and `anBattery`, sub-buckets: `match` · `feature` · `clone` · `key` · `memo` · +`trail` · `equality` · explicit remainder. + +`match` and `feature` come from the `SIL.Machine` layer. To avoid double counting, the HermitCrab +side reads `NodeCostProbe.MatcherTicks` / `FeatureTicks` **before and after** the cascade call and +records the delta — so the lower-layer buckets are slices *inside* the HermitCrab bucket, not +alongside it. Matcher self-time has nested work subtracted at the source (inside `Matcher.cs`); +feature work needs a re-entrancy depth guard or recursive `Unify` calls double-count. + +Measured outcome, so a rebuild can sanity-check itself: on Amharic `anCascade`, clone **46.7%**, +remainder **37.4%**, match 12.5%, feature 3.4%, key **0.0%**, and memo / trail / equality each +**<0.2%**. + +### Censuses + +Rule attempts, split by which guard rejected them (`guardSelector`, +`guardApplicationCount`, `guardUnifiability`, `reachedMatching`); allomorph pattern-match +attempts vs matches; clone created vs discarded vs kept. + +Die points, as an enum with a counter array **and a parallel tick array** — timing each individual +guard check, not the enclosing `Apply()`. Count and cost diverge: `RuleNotApplicableOrPatternMismatch` +was **100.0% of events** and about **0.2% of time**, because each rejection is one list index plus one +reference compare (~29 ns). Never report a die-point histogram by count alone. + +Die points, classified for the abstract-replay question — **shape-free**: synthesis-side +`MaxApplicationCount`, syntactic-feature unification, MPR features, realizational +subsumption/blocking, co-occurrence and obligatory features in `IsWordValid`. **Shape-dependent**: +allomorph pattern match / rule-not-applicable, allomorph environments, final `IsMatch`. +`LexicalLookupMiss` is neither — it precedes any synthesis step. + +--- + +## 2. The keys + +### `AnalysisStateKey` (pre-existing) + +`(Shape, Stratum, SyntacticFeatureStruct, RealizationalFeatureStruct, NonHeadCount, per-rule +un-application count multiset)`. Order-independent by design — no analysis-side rule reads trail +order, which is the redundancy it collapses. The multiset is XOR-combined, not `*31`-rolled, so it +hashes commutatively. + +**Freeze-on-read in the constructor is mandatory.** `AnalysisAffixTemplateRule.Apply` reassigns +`SyntacticFeatureStruct` to a fresh unfrozen clone *after* the owning `Word` is frozen, and that +setter has no `CheckFrozen()` guard. + +### The narrowed variant (row 4) + +Filter the count multiset to rules that are **not** provably shrinking. Two requirements learned the +hard way: build the retain-map **once per grammar**, not per key (key construction is on the hottest +path); and filter during hashing and equality rather than by materialising a filtered dictionary, so +the narrow key allocates no more than the full one. **Absent from the map means retain** — a rule the +classifier walk does not cover must be counted, never silently dropped. + +A `retainedCount` field is needed so equality can compare sizes without enumerating. + +### `SynthesisStateKey` (row 5) — and the trap + +Everything a synthesis step reads: shape+annotations, syntactic FS, realizational FS, MPR set, root +allomorph, disjunctive allomorph indices, applied-rule counts, `IsPartial`, +`IsLastAppliedRuleFinal`, stratum, **and the ordered remaining trail** — +`Word.MorphologicalRuleTrail` sliced to `[0, PendingTrailPosition]`. + +**The trail content is the whole game.** A key with `PendingTrailPosition` (an integer index) and no +trail content is fine for *measuring* and catastrophic for *skipping work*: two candidates at the +same index with different pending sequences compare equal, and merging them discards one +derivation's continuation. Every large apparent redundancy in this engine was measured with such a +key, and each collapsed when the content was added — 9,774x → 15–40%, 61-of-395,026 → 351,414-of-395,026, +3.22x/8.10x → `hits = 0`. + +**`Word.ValueEquals` is not a starting point.** It omits `_syntacticFS`, MPR features and the +disjunctive allomorph indices, all of which synthesis reads. Inheriting from it produces a silently +inflated ratio. + +Two more properties a real memo needs: stored results are **sets**, not values (realizational rules +are trail-exempt — `SynthesisRealizationalAffixProcessRule` has no `IsMorphologicalRuleApplicable` +gate — and several allomorphs of one rule can legitimately match one input before the disjunctive +break); and stored outputs embed the producing candidate's trail, so replay must **re-anchor**, the +mirror of `Word.ReplayOnto`. + +### `RuleLengthClassifier` (row 4's static half) + +Verdicts: `Shrinking` / `NonShrinking` / `Unknown`, where `Unknown` is treated identically to +`NonShrinking` by the key and kept distinct only so a census can tell "this grammar has zero +morphemes" from "this grammar has constructs we cannot analyse". + +An `AffixProcessAllomorph` shrinks iff **every** allomorph (a) inserts at least one *segment* +(`InsertSegments` counting only `HCFeatureSystem.Segment` nodes — a boundary-only insertion is +conservatively NonShrinking, since `InsertSegments.GenerateAnalysisLhs` omits boundaries from the +analysis pattern; `InsertSimpleContext` counts as 1), (b) has **every Lhs part captured** by a Rhs +`CopyFromInput` or `ModifyFromInput`, and (c) copies no part twice. + +The direction trap, which is easy to invert: on the analysis side rules are *un*-applied, so a rule +that inserts on synthesis **removes** on analysis, and a rule that truncates on synthesis +**untruncates** — an Lhs part with no Rhs copy makes the un-applied word **grow**. The authority is +`AnalysisMorphologicalTransform.GenerateShape`, which walks the Lhs parts and either copies the +captured span or calls `Untruncate` to regenerate the part from its pattern. Reduplication (a part +copied twice) and `CompoundingRule` are `Unknown`; conservatism only costs pruning opportunity. + +Measured census, for a rebuild to check against: Sena 19 of 27 rules shrinking, Indonesian 10 of 15, +Amharic 31 of 36. + +--- + +## 3. Harness design + +`[TestFixture]`, `[Explicit]`, environment-driven, writing only `TestContext` lines and no files — +this repo never commits real grammars, and derived corpus data must not leak into a committed path. +**Never print rule, stratum, morpheme or lexical-entry names, or word lists.** Counts, ratios and +timings only. Conformance fixture IDs are committed and safe. + +Env: `HC_MEMO_GRAMMAR`, `HC_MEMO_WORDS`, `HC_MEMO_MAX_WORDS`, `HC_PROBE_WORDS` (explicit +comma-separated list), `HC_PROBE_FIXTURES_ROOT`, `HC_MEMO_TIMEOUT_MS`. + +Breadth comes from `SIL.Machine.Morphology.HermitCrab.Conformance.Fixture.DiscoverAll(root)` over +`conformance/{languages,edge-cases}` — 33 grammars, each `grammar.xml` + `words.yaml` with +hand-derived expectations. Depth comes from the three real corpora. **Amharic: never exceed 28 +words** — word 29 (`ሌባዎቹ`) is pathological and burns hours. + +Run the memoized path: `new Morpher(traceManager, language, maxDegreeOfParallelism: 1)`. Tracing +disables the memo, so a traced run measures nothing. + +### A/B methodology, all of it earned by being wrong first + +Parity is the gate and speed is only reported. Acceptance is **analysis-set equality on canonical +morpheme-signature sets** (`MorpherTests.WordAnalysisSignature`, sorted) — never object or byte +equality, because a replayed `Word` is legitimately not field-for-field identical. + +Timing: **discard a warm-up rep per arm, then take min of N interleaved samples, and print the +off-arm spread beside every speedup.** Refuse any speedup smaller than its own spread. Three +failures forced each part of this — a first A/B read 1.53x that was entirely JIT; another read +2.385x with a 210.9% spread on one arm that was 1.015x serial; and adding one arm's warm-up while +discarding the other's manufactured a uniform ~2.4x across every fixture including ones with zero +shareable steps. + +**Never dispatch timing-sensitive probes concurrently.** Three at once put four testhosts on the +machine and invalidated three rows' bucket percentages. Check `tasklist | grep -c testhost` before +any timing run and report the count alongside the numbers. Counts survive contention; shares and +speedups do not. + +**Never aggregate the test logger's console output.** It prints every line twice, the second copy +space-prefixed, so a `^\s*` grep doubles every total — this published a 2x error. + +Sub-50 ms fixtures cannot carry a timing claim: one 10 ms fixture's share moved 1.3% → 14.5% +between two runs of identical code. Flag them (`reliable = wallMs >= 50`) rather than filtering them +out silently. + +### One concurrency hazard worth knowing + +A per-word timeout wrapper built on `Task` has no cooperative cancellation, so an abandoned word's +task keeps mutating engine state after the harness moves on. Harmless with a per-word +`AnalysisScope`; under a **shared** scope it collided with the next word on the same +`Memo`/`InProgress` collections and produced `IndexOutOfRangeException` inside +`MemoizedCombinationRuleCascade`. A shared-arm timeout must abort the run, not continue. Also note +`Task.Result` throws `AggregateException`, not the bare exception type. + +--- + +## 4. Two durable engine facts, restated here so they survive independently + +**`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and `_nonHeadApps` +only. This is safe *solely* because `AnalysisStateKey` includes the count multiset, which guarantees +a memo hit has identical arrival and stored-arrival counts. **Narrowing that key breaks correctness +with no test failure.** Fix, if the key ever changes: store arrival counts on `MemoEntry` and compute +`stored - storedArrival + query` — a no-op under today's key, which is also how to test it. + +**`AnalysisScope.MaxMemoEntries` is a per-scope cap with no eviction.** Sharing one scope across a +corpus therefore makes words compete for a fixed budget: two heavy Sena words exhausted it at index +74 and 76, and every later word lost the within-word memoization a fresh scope would have given it. +Cross-word hits went *negative* and the corpus ran 28–32% slower. A shared bounded cache without +eviction is not a smaller version of the optimization.