From 6b5b07e43222026f85ed9e924b0e05d11464571f Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Tue, 25 Aug 2026 16:30:40 -0500 Subject: [PATCH 1/2] backlog: file #1357 -- the connscale FD probe cannot see a multi-process engine A single probe tick costs longer than the entire measurement window, so the PID set is resolved once per sweep step and never re-resolved. Measured: one tick costs 1419-1848 ms against hold_seconds = 1.5. _RESOLVE_EVERY_TICKS = 8 is therefore unreachable by construction -- not a badly-chosen constant, one the window cannot afford at any cadence. Its own comment reasons at "the runner's poll cadence", which this profile does not have. The consequence is the reason it is worth a row: nothing on this project can currently measure the resource posture of a multi-process engine. That is what holds #1278, rather than any test failure. The item carries the per-tick trace, because the summary number hides the finding. The walk's PID set varies 2, 3, 8, 50 across CONSECUTIVE TICKS OF ONE RUN and handles track it almost linearly -- so handles_peak measures what the walk caught, not what the engine holds. Four things it is not, each checked rather than assumed: not probe degradation (zero degraded ticks, and degradation makes that SLO pass vacuously so it cannot red it); not the #220 PID-set gate (that predicate is on the CPU path, the FD gauge is a plain max); not PR 598 (grepped against its diff -- and its test constructs FdSampler(resolve_every=1) explicitly, so the fixed guard PASSES against this defect, proving re-resolution works when asked while production never asks); and not a loose SLO (a red needs a collapse below 75 percent). It also deliberately does NOT carry a resource figure. Forcing per-tick resolve produced ~3745 handles against a ~385 baseline, which would be ~312 per worker child. That number is withdrawn and the item says so in terms, because the trace shows the denominator is not stable enough to divide by. Leaving a withdrawn figure quotable from a backlog row is how it comes back as a fact. Number allocated with scripts/coord/alloc.ps1 and the row added in the same commit. backlog_status_check passes: 600 items, each declaring exactly one status. Co-Authored-By: Claude Opus 5 --- docs/BACKLOG.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 80f3cc21..fbd07be0 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -16120,3 +16120,38 @@ federation on, the authorization request would travel the browser-visible front identity provider is the only party positioned to reject tampering. That is true, and it is not by itself a finding against this cell. + +## 1357. The connscale FD probe cannot see a multi-process engine: PID-set re-resolution never runs + +> 🔢 **Filed 2026-08-25 - not started.** ***A single probe tick costs LONGER THAN THE ENTIRE MEASUREMENT WINDOW, so the PID set is resolved once per sweep step and never re-resolved -- and a multi-process engine's worker children are invisible to the FD gauge.*** **Measured: one tick costs 1419-1848 ms against `hold_seconds = 1.5`.** So `_RESOLVE_EVERY_TICKS = 8` ([`probe.py:60`](../harness/load/connscale/probe.py)) is **unreachable by construction** -- not a badly-chosen constant, a constant the window cannot afford at any cadence. Its own comment reasons at "the runner's poll cadence", which this profile does not have. +> Verdict: build +> Research: none +> Closing-act: code + +**Cluster:** test harness / measurement. **Priority:** P2. **Verdict:** build. +**Severity:** no product axis (sec. 0) -- this is an instrument, not shipped engine behaviour. ***The cost is that NOTHING ON THIS PROJECT CAN CURRENTLY MEASURE THE RESOURCE POSTURE OF A MULTI-PROCESS ENGINE***, which is what holds BACKLOG #1278 rather than any test failure. + +**THE EVIDENCE, one run, every tick traced with its cost and PID-set size:** + +| handles | pids | tick cost | +|---|---|---| +| 1266 | 8 | 1419 ms | +| 543 | 3 | 1528 ms | +| 7118 | 50 | 1732 ms | +| 406 | 2 | 1483 ms | +| 479 | 2 | 1848 ms | + +**The walk's PID set varies 2, 3, 8, 50 ACROSS CONSECUTIVE TICKS OF ONE RUN, and handles track it almost linearly.** So `handles_peak` is not measuring the engine's footprint -- **it is measuring how many processes the walk happened to catch.** 50 pids for a 24-inbound engine is roughly double the 25 processes the design predicts, so the walk is plausibly crediting sink or harness processes. + +**HOW IT SURFACES.** Flipping `[sandbox].mode` to `subprocess` spawns one worker child per inbound. `tests/test_connscale_smoke.py::test_the_fd_and_empty_claim_curves_are_monotonic_in_n` then fails **4 runs in 5**, against **0 in 5** at `mode=off` (controlled A/B, same worktree, same tests, nothing else running; Fisher exact two-tailed p = 0.0476). Under `off` the count is near-noiseless -- ~385 at N=12 rising to ~421 at N=24 -- because with no children an engine-only PID set is COMPLETE. + +**FOUR THINGS THIS IS NOT, each checked rather than assumed:** + +1. **NOT probe degradation.** `fd_probe_degraded_ticks = 0` and `fd_probe_degraded = ()` on every record, every run, both arms, including every failing run. And it could not have mattered: [`runner.py:1224-1234`](../harness/load/connscale/runner.py) has `if val is None: continue` **above** `prev_val = v`, so a degraded reading is deleted and can never be an endpoint -- degradation makes this SLO pass **vacuously**, it cannot red it. +2. **NOT the `#220` PID-set gate.** That predicate lives only in the `cpu_readings` comprehension; the FD gauge at `:1071` is a plain `max(handles)` with no PID-set predicate. +3. **NOT [PR 598](https://github.com/MEFORORG/MessageFoundry/pull/598).** Verified by grep against its diff: it touches neither `_RESOLVE_EVERY_TICKS` nor its call sites. That fix is about re-resolution producing the wrong VERDICT when it runs; this is re-resolution NOT RUNNING. **Worse, 598's test constructs `FdSampler(os.getpid(), resolve_every=1)` explicitly, so the fixed guard PASSES against this defect** -- it proves re-resolution works when asked, and production never asks. +4. **NOT a loose SLO tipping over.** `_MONOTONIC_TOLERANCE = 0.25`, so a red needs the larger-N reading **below 75 percent** of the smaller-N one. That is a collapse, not jitter. + +**A FIGURE THIS ITEM DELIBERATELY DOES NOT CARRY.** Forcing `resolve_every=1` produced readings of ~3745 handles against a ~385 baseline, which would be ~312 handles per worker child. **That number is WITHDRAWN and must not be quoted from here:** the trace above shows the denominator is not stable enough to divide by. The resource question is real and open; this item is the reason nobody can answer it yet. + +**Reproducer:** the per-tick trace and the A/B harness live in the filing session's scratchpad (`fdprobe_ab.py`); it reuses `test_connscale_smoke`'s own fixture setup and takes ~25 s a run. Rebuilding it is a few lines against `run_connscale`. From 8fea0a88b53699557baed081520ee167f9e5d31a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Wed, 26 Aug 2026 09:53:23 -0500 Subject: [PATCH 2/2] backlog: complete #1357's limb-B chain -- the degraded reading never reaches a comparison at all The row already said runner.py drops a None reading so degradation makes that SLO pass vacuously. It did not say where the None comes from, which left the chain open at the end whoever fixes this would start from. Both halves, verified against both files: probe.py:165 every degrade site returns through _gap(), which sets handles=None -- so a degraded tick IS a None reading runner.py:1225 `if val is None: continue` runs BEFORE v is computed and WITHOUT updating prev_val -- so such a reading can never be EITHER endpoint of a comparison, not merely a skipped one The severity distinction is worth carrying too: limb A is wrong today, while limb B is a guard that cannot fail until its trigger fires. Those want different urgency and the row now says so. The _gap() half is the Dispatcher's reading and is attributed in the row. I verified it here rather than taking it -- the row's whole value is that it is measured, and a borrowed link would be the one unmeasured claim in it. Co-Authored-By: Claude Opus 5 --- docs/BACKLOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index d06ab28d..90653099 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -16289,7 +16289,7 @@ catch. **FOUR THINGS THIS IS NOT, each checked rather than assumed:** -1. **NOT probe degradation.** `fd_probe_degraded_ticks = 0` and `fd_probe_degraded = ()` on every record, every run, both arms, including every failing run. And it could not have mattered: [`runner.py:1224-1234`](../harness/load/connscale/runner.py) has `if val is None: continue` **above** `prev_val = v`, so a degraded reading is deleted and can never be an endpoint -- degradation makes this SLO pass **vacuously**, it cannot red it. +1. **NOT probe degradation.** `fd_probe_degraded_ticks = 0` and `fd_probe_degraded = ()` on every record, every run, both arms, including every failing run. And it could not have mattered, for a reason that runs end to end: EVERY degrade site in the probe returns through `_gap()` ([`probe.py:165`](../harness/load/connscale/probe.py)), which sets `handles=None`; and [`runner.py:1225-1226`](../harness/load/connscale/runner.py) does `if val is None: continue` **before** `v` is computed and **without updating `prev_val`**. So a degraded reading is not merely skipped -- it can never be either endpoint of a comparison. **Degradation makes this SLO pass VACUOUSLY; it cannot red it.** That is a guard which cannot fail until its trigger fires, and it is a different severity from limb A, which is wrong today. *(The `_gap()` half of this chain is the Dispatcher's reading, verified here against both files.)* 2. **NOT the `#220` PID-set gate.** That predicate lives only in the `cpu_readings` comprehension; the FD gauge at `:1071` is a plain `max(handles)` with no PID-set predicate. 3. **NOT [PR 598](https://github.com/MEFORORG/MessageFoundry/pull/598).** Verified by grep against its diff: it touches neither `_RESOLVE_EVERY_TICKS` nor its call sites. That fix is about re-resolution producing the wrong VERDICT when it runs; this is re-resolution NOT RUNNING. **Worse, 598's test constructs `FdSampler(os.getpid(), resolve_every=1)` explicitly, so the fixed guard PASSES against this defect** -- it proves re-resolution works when asked, and production never asks. 4. **NOT a loose SLO tipping over.** `_MONOTONIC_TOLERANCE = 0.25`, so a red needs the larger-N reading **below 75 percent** of the smaller-N one. That is a collapse, not jitter.