Skip to content

feat(control): restart-free fast path for /apply on watchdog-only keys - #397

Merged
VijitSingh97 merged 2 commits into
developfrom
feat/381-apply-fast-path
Aug 22, 2026
Merged

feat(control): restart-free fast path for /apply on watchdog-only keys#397
VijitSingh97 merged 2 commits into
developfrom
feat/381-apply-fast-path

Conversation

@VijitSingh97

Copy link
Copy Markdown
Contributor

Summary

POST /apply's privileged applier (control_apply in rigforge.sh) used to re-run the entire
apply pipeline for every accepted change — regenerate XMRig's config, re-render its unit, restart
the service, then poll for a live pool connection — even for a change that never touches XMRig at
all. A live walkthrough (#344 item 1) measured a single watchdog_interval_min change taking ~62s
round-trip because of this. This closes the gap with a restart-free fast path for the two keys
that are provably restart-free from the applier's own code, while leaving every other key on the
existing full, XMRig-restarting path unchanged.

Per the #344 discussion, the fast path must not fork a second apply implementation that can drift
from the real one: it reuses install_watchdog — the exact call the full apply() pipeline already
makes on every run — instead of re-implementing unit rendering, and it stamps config_meta
provenance the same way apply() does, so a consumer reading the feed cannot tell which path served
a change.

Update after an independent security review of the first version of this PR: the review found a
HIGH finding in _control_do_apply_fast's original success check (a bare systemctl is-active) —
see "Security review fix" below. That's fixed in this PR; the review also surfaced a related,
pre-existing hazard on develop today (the full apply path force-restarts a rig that is
deliberately stopped) that is out of scope for this PR and is filed separately.

Mechanism

  • CONTROL_FAST_PATH_KEYS="watchdog_interval_min max_temp_c" (rigforge.sh) is a closed
    allowlist
    . _control_fast_path_eligible() checks it as a subset match — every key in the
    committed change must be on the list — never a "not on the slow list" complement. An empty,
    malformed, or jq-failure-sentinel ("?") keys-csv also fails closed to "not eligible". This means
    any future addition to CONTROL_WRITABLE_KEYS that nobody has re-proven restart-free here
    automatically takes the full path, by construction — nothing has to remember to update a second
    list.

  • _control_do_apply_fast() runs when control_apply() classifies a change as eligible. It:

    1. parse_config (refresh the just-committed values),
    2. install_watchdog (re-renders + daemon-reloads the watchdog timer — a no-op write if the
      cadence didn't change, which is also what the full path already does unconditionally),
    3. _stamp_config_meta with the same RIGFORGE_CONFIG_SOURCE/RIGFORGE_CONFIG_CHANGE_ID
      dynamic-scope values apply() itself uses, and
    4. succeeds iff the miner's run-state did not degrade — see "Security review fix" below; it is
      not a bare systemctl is-active check.

    It deliberately skips apply()/_apply_runtime entirely: no generate_xmrig_config, no XMRig
    unit re-render, no systemctl restart of the miner, no _wait_miner_live retry loop (which alone
    is up to 20 × 3s = 60s), no _apply_pool_check.

  • control_apply() now dispatches on _control_fast_path_eligible before calling either
    _control_do_apply_fast or the existing _control_do_apply. A failure of either path falls
    through to the exact same rollback branch that already existed (restore the pre-change snapshot,
    re-apply through the full path, record rolled_back/failed) — a wrong "eligible" verdict, or the
    fast path failing for an unrelated reason, can never leave a change silently stuck or misreported
    as applied.

  • No new endpoint, no change to bearer auth, no change to what the control path accepts
    (util/control-server.py's WRITABLE set and safety backstops are untouched) — this only changes
    how rigforge.sh control-apply executes an already-accepted, already-validated change.

Security review fix: run-state, not is-active alone

The first version of _control_do_apply_fast gated success solely on systemctl is-active --quiet "$SERVICE_NAME" after the watchdog reconcile. That's wrong for a rig that is legitimately
stopped
when the fast-path change lands — a watchdog thermal hold (watchdog() stops the miner and
leaves a marker so it knows to stay stopped), or an operator's manual stop. The fast path never
touches the XMRig unit or service, so a stopped rig staying stopped is correct, not a failure — but
the old check would have reported it as one, and control_apply's rollback branch would then cp
the pre-change backup over config.json (discarding the operator's change) and call
_control_do_applyapply()_apply_runtime, which does an unconditional
sudo systemctl restart — force-restarting a rig that was deliberately offline. Worst case: changing
max_temp_c during a thermal hold, which is exactly the headline use case this fast path exists for.

Fix (rigforge.sh, _control_do_apply_fast): capture the service's is-active state before the
watchdog reconcile, and again after. Success is "the run-state did not degrade":

  • inactive-before → success, regardless of after (it was never this change's job to start or keep
    the miner running; the fast path can't have caused whatever state it's in either way; the new
    value takes effect on the watchdog's own next tick).
  • active-before, active-after → success (the original, unchanged happy path).
  • active-before, inactive-after → failure → the existing rollback (a real regression).

No thermal-hold-marker special-casing — the before/after comparison covers every "was already down"
case with less machinery, per the review's minimal-fix guidance.

Pre-existing, out-of-scope hazard the review also surfaced (filed separately, not in this PR):
_apply_runtime's sudo systemctl restart is unconditional for the full apply path and for the
rollback re-apply too — it doesn't know about a thermal hold or a manual stop either. That means a
full-path control-apply change (any key outside this PR's fast-path allowlist), or any rollback
(including a fast-path change that failed for an unrelated reason), still force-restarts a thermally
held or manually stopped rig today, independent of #381. This PR does not touch _apply_runtime or
the full/rollback path — see the filed-separately issue draft for the mechanism and fix directions.

The allowlist and why each key is restart-free

Evidence is from rigforge.sh itself, not asserted:

  • watchdog_interval_min — bakes into only rigforge-watchdog.timer's OnUnitActiveSec,
    rendered by install_watchdog. generate_xmrig_config (XMRig's own generated config) and the
    XMRig unit template never reference it — confirmed by grepping both for the key. Reusing
    install_watchdog (the same call the full apply() already makes unconditionally on every run) is
    therefore sufficient: it re-renders the timer with the new cadence and daemon-reloads it. XMRig
    itself has nothing to reload.
  • max_temp_c — never rendered into any unit. install_watchdog's own comment (predating this
    issue) already says why: "Only the cadence is baked into the units — the verb re-reads
    config.json every run, so an apply after a max_temp_c ... edit needs no unit rewrite." The
    watchdog verb picks up the new value the next time its timer fires; nothing beyond the
    config.json write _control_commit already performs is needed.

Both keys were named as "the obvious candidates" in the original #344/#381 issue text and are
verifiable straight from the code above. I did not add any other CONTROL_WRITABLE_KEYS member
(pools, DONATION, autotune, watchdog) to the fast path:

  • pools and DONATION are written into XMRig's own generated config (generate_xmrig_config) —
    a restart is the only way XMRig serves the new values, so these definitively stay on the full path.
  • autotune and the watchdog enable/disable flag each govern an install_* call this change did
    not audit for restart-freedom (in watchdog's specific case, the remote control path can only ever
    turn it on, per the existing Control path: safety-critical changes (watchdog disable / max_temp_c) apply silently — rollback checks liveness, not thermal protection #257 safety backstop, which would mean the very first transition
    needs install_watchdog to create — not just re-render — the timer/service pair; I did not verify
    that transition is safe to run without the surrounding full-pipeline reconciliation, so it stays on
    the full path). If either is proven restart-free later, extending CONTROL_FAST_PATH_KEYS is a
    one-line change with its own evidence trail, per the closed-allowlist design.

Files changed

  • rigforge.sh:
    • CONTROL_FAST_PATH_KEYS, _control_fast_path_eligible(), _control_do_apply_fast() — new,
      inserted after _control_do_apply().
    • control_apply() — dispatches on _control_fast_path_eligible instead of unconditionally
      calling _control_do_apply; the failure/rollback branch is otherwise unchanged.
  • tests/run.sh:
    • _control_fast_path_eligible classification tests (== unit: _control_fast_path_eligible — closed allowlist classification (#381) ==).
    • _control_do_apply_fast integration test against a real install_watchdog render (== unit: _control_do_apply_fast — reuses install_watchdog, skips apply()/xmrig restart (#381) ==).
    • ca_exec's harness (== unit: control_apply orchestration + rollback (#236) ==) extended with
      full-apply-called/fast-apply-called marker files and a _control_do_apply_fast stub, plus new
      control_apply() dispatch/fallback assertions.
    • New caf_exec/caf_run harness with a stateful systemctl stub (== unit: control_apply + REAL _control_do_apply_fast — run-state criterion, not is-active alone (#381 security review) ==) — runs control_apply end to end with the real _control_do_apply_fast body (not
      stubbed), since the fix under review is inside that function.
  • docs/adr/0001-writable-worker-config-control-path.md — new D12 decision record, updated to
    state the run-state criterion (not the original is-active-alone claim).
  • docs/operations.md, docs/pithead-integration.md, docs/configuration.md — note the fast path
    where they already describe /apply's behavior and cost.
  • CHANGELOG.md[Unreleased] entry.

Tests — what each one catches

All in tests/run.sh, dependency-free (bash tests/run.sh / make test):

_control_fast_path_eligible (pure classification):

Assertion Mutation it catches
watchdog_interval_min / max_temp_c alone → eligible; both together → eligible Classification inverted (return 0/return 1 swapped)
pools, DONATION, autotune, watchdog alone → not eligible Classification inverted, or the allowlist silently including a full-path key
DONATION,max_temp_c (mixed) → not eligible The "ALL keys must qualify" subset check loosened to an "ANY key" check
some_future_key alone → not eligible The closed-set subset check inverted into a "not on the slow list" complement
empty / "?" → not eligible The fail-closed default on malformed input removed
CONTROL_FAST_PATH_KEYS extracted from the script == exactly {max_temp_c, watchdog_interval_min} The allowlist silently growing (or shrinking) without a matching evidence-trail/test update
fast-path allowlist ⊆ control-writable allowlist A fast-path key added without the corresponding CONTROL_WRITABLE_KEYS entry (dead code)

_control_do_apply_fast (against a real install_watchdog render):

Assertion Mutation it catches
returns 0 when the miner service is active The success path miscomputing its own return code
apply() marker file never written The fast path accidentally calling the full, XMRig-restarting pipeline
rendered timer's OnUnitActiveSec reflects the new interval install_watchdog not actually being invoked, or invoked with stale values
config_meta.source == control, last_change_id == the test's cid The provenance stamp being dropped on the fast path (a real dashboard-facing regression, not just a latency one)

control_apply() + the REAL _control_do_apply_fast, against a stateful systemctl stub that
answers the two is-active calls differently (the generic always-succeeds stub used everywhere else
can't exercise this — that gap is exactly what the review flagged):

Assertion Mutation it catches
inactive-before, inactive-after → status applied, new value lands in config.json, no rollback/restart attempted Reverting the run-state comparison back to a naive "is it active now" check — that mutant sees "not active" and wrongly rolls the change back, discarding it and force-restarting a deliberately-stopped rig
active-before, inactive-after → status rolled_back, config restored, rollback re-apply invoked The run-state criterion being dropped entirely (e.g. always returning success) — a genuine regression must still trip the rollback

control_apply() orchestration (extends the existing #236 rollback harness with
full-apply-called/fast-apply-called markers):

Assertion Mutation it catches
watchdog_interval_min-only / max_temp_c-only / both-together → fast marker set, full marker absent, status applied The dispatch calling the wrong function, or calling both
DONATION,max_temp_c (mixed) → full marker set, fast marker absent The ALL-keys-must-qualify check loosened to ANY-key (would let a DONATION change skip its required restart)
DONATION-only → full marker set, fast marker absent The full path being bypassed for existing, well-covered writable keys
fast-path failure (CA_FAST_APPLY_OK=0) → status rolled_back, config restored, rollback re-apply uses the full path The fast-path failure branch skipping the rollback, or reporting applied regardless of the fast apply's return code

What was run

$ make lint
shellcheck --severity=warning rigforge.sh tests/coverage.sh tests/e2e-pithead.sh tests/e2e-real.sh \
  tests/e2e/in-container.sh tests/e2e/linux.sh tests/e2e/macos.sh tests/run.sh tests/smoke.sh \
  util/proposed-grub.sh
shfmt -i 4 -d rigforge.sh tests/coverage.sh tests/e2e-pithead.sh tests/e2e-real.sh \
  tests/e2e/in-container.sh tests/e2e/linux.sh tests/e2e/macos.sh tests/run.sh tests/smoke.sh \
  util/proposed-grub.sh
# clean, no output from either tool

$ bash tests/run.sh
rigforge tests: 1799 passed, 1 failed

$ npx --yes markdownlint-cli2@0.22.1 docs/adr/0001-writable-worker-config-control-path.md \
  docs/operations.md docs/pithead-integration.md docs/configuration.md CHANGELOG.md
Summary: 0 error(s)

The 1 failure (msr-apply: missing wrmsr warns, never fails the unit (#140)) is a pre-existing,
unrelated environment flake: this dev box has a real /usr/sbin/wrmsr binary that leaks into that
test's PATH stub, so it emits "Permission denied" instead of the expected "not found" — already
tracked with its own fix (test: pin the missing-wrmsr case's PATH so the host's real wrmsr cannot leak in). Every assertion this PR adds passed; re-ran with grep -c "(#381)" isolated to confirm
none were red.

Not run: make test-e2e (container e2e), make e2e-real/make e2e-pithead (real-rig/bench gates) —
this change is unit/repo-level only, per the task scope; no real miner or bench hardware was touched.

Follow-up filed separately (not in this PR)

The pre-existing hazard described under "Security review fix" above (_apply_runtime's unconditional
restart on the full apply/rollback path, independent of #381) is drafted as its own issue and will be
filed after this PR — deliberately out of scope here to keep this change to the fast path it was
asked to add.

Closes #381.

control_apply re-ran the entire apply() pipeline for every accepted
change -- regenerate XMRig's config, re-render its unit, restart the
service, then poll for a live pool connection -- even for a change
that never touches XMRig at all. A live walkthrough (#344 item 1)
measured a single watchdog_interval_min change taking ~62s round-trip
because of this.

CONTROL_FAST_PATH_KEYS is a closed allowlist (watchdog_interval_min,
max_temp_c), checked as a subset match so an unrecognised key --
including any future CONTROL_WRITABLE_KEYS addition nobody has
re-proven restart-free here -- takes the full path by construction.
Both keys are proven restart-free from generate_xmrig_config and
install_watchdog's own comments, not asserted. The fast path reuses
install_watchdog verbatim (the same call apply() already makes on
every run) instead of re-implementing unit rendering, per the #344
constraint that this must not fork a second apply implementation that
can drift from the real one; it stamps config_meta provenance the same
way apply() does. A fast-path failure falls through to the same
full-pipeline rollback a failed full apply already uses.

Closes #381.
…e alone (#381)

An independent security review of the fast path found a HIGH-severity
gap: gating success on a bare `systemctl is-active` reads a
legitimately stopped rig -- a watchdog thermal hold, or an operator's
manual stop -- as a fast-path failure. control_apply's rollback then
discards the operator's change and calls the full apply() pipeline,
which unconditionally restarts the service, force-restarting a rig
that was deliberately offline. Worst case: changing max_temp_c during
a thermal hold, the exact case this fast path exists for.

Fix: capture is-active before the watchdog reconcile and again after.
Success is "the run-state did not degrade" -- inactive-before is
success regardless of after (the fast path never touches the xmrig
unit, so it can't have caused whatever state the rig is in either
way); only active-before/inactive-after is a real regression and
still rolls back. No thermal-hold-marker special-casing needed.

Tests use a stateful systemctl stub (the generic always-succeeds stub
elsewhere never exercised the inactive branch) driving control_apply
end to end through the real _control_do_apply_fast body.

ADR 0001's D12 paragraph updated to state the run-state criterion in
place of the original is-active-alone claim.
@VijitSingh97
VijitSingh97 merged commit 8de186a into develop Aug 22, 2026
9 checks passed
@VijitSingh97
VijitSingh97 deleted the feat/381-apply-fast-path branch August 22, 2026 20:09
VijitSingh97 added a commit that referenced this pull request Aug 23, 2026
Restart-free control-apply fast path (#397/#381) and the HugePages pool-ceiling
contract that closes the co-resident double-count (#399/#398, pithead#1103).
Recovers the topology guard, e2e-dashboard leg, and msr-test PATH fix that had
landed on main only (back-merge restoring the main-ancestor invariant).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U189N8GtbdUVtm8UtVNcDw
@VijitSingh97 VijitSingh97 mentioned this pull request Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control /apply: fast path for restart-free keys (from #344 item 1)

1 participant