feat(control): restart-free fast path for /apply on watchdog-only keys - #397
Merged
Conversation
control_apply re-ran the entire apply() pipeline for every accepted change -- regenerate XMRig's config, re-render its unit, restart the service, then poll for a live pool connection -- even for a change that never touches XMRig at all. A live walkthrough (#344 item 1) measured a single watchdog_interval_min change taking ~62s round-trip because of this. CONTROL_FAST_PATH_KEYS is a closed allowlist (watchdog_interval_min, max_temp_c), checked as a subset match so an unrecognised key -- including any future CONTROL_WRITABLE_KEYS addition nobody has re-proven restart-free here -- takes the full path by construction. Both keys are proven restart-free from generate_xmrig_config and install_watchdog's own comments, not asserted. The fast path reuses install_watchdog verbatim (the same call apply() already makes on every run) instead of re-implementing unit rendering, per the #344 constraint that this must not fork a second apply implementation that can drift from the real one; it stamps config_meta provenance the same way apply() does. A fast-path failure falls through to the same full-pipeline rollback a failed full apply already uses. Closes #381.
…e alone (#381) An independent security review of the fast path found a HIGH-severity gap: gating success on a bare `systemctl is-active` reads a legitimately stopped rig -- a watchdog thermal hold, or an operator's manual stop -- as a fast-path failure. control_apply's rollback then discards the operator's change and calls the full apply() pipeline, which unconditionally restarts the service, force-restarting a rig that was deliberately offline. Worst case: changing max_temp_c during a thermal hold, the exact case this fast path exists for. Fix: capture is-active before the watchdog reconcile and again after. Success is "the run-state did not degrade" -- inactive-before is success regardless of after (the fast path never touches the xmrig unit, so it can't have caused whatever state the rig is in either way); only active-before/inactive-after is a real regression and still rolls back. No thermal-hold-marker special-casing needed. Tests use a stateful systemctl stub (the generic always-succeeds stub elsewhere never exercised the inactive branch) driving control_apply end to end through the real _control_do_apply_fast body. ADR 0001's D12 paragraph updated to state the run-state criterion in place of the original is-active-alone claim.
VijitSingh97
added a commit
that referenced
this pull request
Aug 23, 2026
Restart-free control-apply fast path (#397/#381) and the HugePages pool-ceiling contract that closes the co-resident double-count (#399/#398, pithead#1103). Recovers the topology guard, e2e-dashboard leg, and msr-test PATH fix that had landed on main only (back-merge restoring the main-ancestor invariant). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U189N8GtbdUVtm8UtVNcDw
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /apply's privileged applier (control_applyinrigforge.sh) used to re-run the entireapplypipeline for every accepted change — regenerate XMRig's config, re-render its unit, restartthe service, then poll for a live pool connection — even for a change that never touches XMRig at
all. A live walkthrough (#344 item 1) measured a single
watchdog_interval_minchange taking ~62sround-trip because of this. This closes the gap with a restart-free fast path for the two keys
that are provably restart-free from the applier's own code, while leaving every other key on the
existing full, XMRig-restarting path unchanged.
Per the #344 discussion, the fast path must not fork a second apply implementation that can drift
from the real one: it reuses
install_watchdog— the exact call the fullapply()pipeline alreadymakes on every run — instead of re-implementing unit rendering, and it stamps
config_metaprovenance the same way
apply()does, so a consumer reading the feed cannot tell which path serveda change.
Update after an independent security review of the first version of this PR: the review found a
HIGH finding in
_control_do_apply_fast's original success check (a baresystemctl is-active) —see "Security review fix" below. That's fixed in this PR; the review also surfaced a related,
pre-existing hazard on
developtoday (the full apply path force-restarts a rig that isdeliberately stopped) that is out of scope for this PR and is filed separately.
Mechanism
CONTROL_FAST_PATH_KEYS="watchdog_interval_min max_temp_c"(rigforge.sh) is a closedallowlist.
_control_fast_path_eligible()checks it as a subset match — every key in thecommitted change must be on the list — never a "not on the slow list" complement. An empty,
malformed, or jq-failure-sentinel (
"?") keys-csv also fails closed to "not eligible". This meansany future addition to
CONTROL_WRITABLE_KEYSthat nobody has re-proven restart-free hereautomatically takes the full path, by construction — nothing has to remember to update a second
list.
_control_do_apply_fast()runs whencontrol_apply()classifies a change as eligible. It:parse_config(refresh the just-committed values),install_watchdog(re-renders +daemon-reloads the watchdog timer — a no-op write if thecadence didn't change, which is also what the full path already does unconditionally),
_stamp_config_metawith the sameRIGFORGE_CONFIG_SOURCE/RIGFORGE_CONFIG_CHANGE_IDdynamic-scope values
apply()itself uses, andnot a bare
systemctl is-activecheck.It deliberately skips
apply()/_apply_runtimeentirely: nogenerate_xmrig_config, no XMRigunit re-render, no
systemctl restartof the miner, no_wait_miner_liveretry loop (which aloneis up to 20 × 3s = 60s), no
_apply_pool_check.control_apply()now dispatches on_control_fast_path_eligiblebefore calling either_control_do_apply_fastor the existing_control_do_apply. A failure of either path fallsthrough to the exact same rollback branch that already existed (restore the pre-change snapshot,
re-apply through the full path, record
rolled_back/failed) — a wrong "eligible" verdict, or thefast path failing for an unrelated reason, can never leave a change silently stuck or misreported
as applied.
No new endpoint, no change to bearer auth, no change to what the control path accepts
(
util/control-server.py'sWRITABLEset and safety backstops are untouched) — this only changeshow
rigforge.sh control-applyexecutes an already-accepted, already-validated change.Security review fix: run-state, not is-active alone
The first version of
_control_do_apply_fastgated success solely onsystemctl is-active --quiet "$SERVICE_NAME"after the watchdog reconcile. That's wrong for a rig that is legitimatelystopped when the fast-path change lands — a watchdog thermal hold (
watchdog()stops the miner andleaves a marker so it knows to stay stopped), or an operator's manual stop. The fast path never
touches the XMRig unit or service, so a stopped rig staying stopped is correct, not a failure — but
the old check would have reported it as one, and
control_apply's rollback branch would thencpthe pre-change backup over
config.json(discarding the operator's change) and call_control_do_apply→apply()→_apply_runtime, which does an unconditionalsudo systemctl restart— force-restarting a rig that was deliberately offline. Worst case: changingmax_temp_cduring a thermal hold, which is exactly the headline use case this fast path exists for.Fix (
rigforge.sh,_control_do_apply_fast): capture the service'sis-activestate before thewatchdog reconcile, and again after. Success is "the run-state did not degrade":
the miner running; the fast path can't have caused whatever state it's in either way; the new
value takes effect on the watchdog's own next tick).
No thermal-hold-marker special-casing — the before/after comparison covers every "was already down"
case with less machinery, per the review's minimal-fix guidance.
Pre-existing, out-of-scope hazard the review also surfaced (filed separately, not in this PR):
_apply_runtime'ssudo systemctl restartis unconditional for the full apply path and for therollback re-apply too — it doesn't know about a thermal hold or a manual stop either. That means a
full-path control-apply change (any key outside this PR's fast-path allowlist), or any rollback
(including a fast-path change that failed for an unrelated reason), still force-restarts a thermally
held or manually stopped rig today, independent of #381. This PR does not touch
_apply_runtimeorthe full/rollback path — see the filed-separately issue draft for the mechanism and fix directions.
The allowlist and why each key is restart-free
Evidence is from
rigforge.shitself, not asserted:watchdog_interval_min— bakes into onlyrigforge-watchdog.timer'sOnUnitActiveSec,rendered by
install_watchdog.generate_xmrig_config(XMRig's own generated config) and theXMRig unit template never reference it — confirmed by grepping both for the key. Reusing
install_watchdog(the same call the fullapply()already makes unconditionally on every run) istherefore sufficient: it re-renders the timer with the new cadence and
daemon-reloads it. XMRigitself has nothing to reload.
max_temp_c— never rendered into any unit.install_watchdog's own comment (predating thisissue) already says why: "Only the cadence is baked into the units — the verb re-reads
config.json every run, so an apply after a max_temp_c ... edit needs no unit rewrite." The
watchdogverb picks up the new value the next time its timer fires; nothing beyond theconfig.jsonwrite_control_commitalready performs is needed.Both keys were named as "the obvious candidates" in the original #344/#381 issue text and are
verifiable straight from the code above. I did not add any other
CONTROL_WRITABLE_KEYSmember(
pools,DONATION,autotune,watchdog) to the fast path:poolsandDONATIONare written into XMRig's own generated config (generate_xmrig_config) —a restart is the only way XMRig serves the new values, so these definitively stay on the full path.
autotuneand thewatchdogenable/disable flag each govern aninstall_*call this change didnot audit for restart-freedom (in
watchdog's specific case, the remote control path can only everturn it on, per the existing Control path: safety-critical changes (watchdog disable / max_temp_c) apply silently — rollback checks liveness, not thermal protection #257 safety backstop, which would mean the very first transition
needs
install_watchdogto create — not just re-render — the timer/service pair; I did not verifythat transition is safe to run without the surrounding full-pipeline reconciliation, so it stays on
the full path). If either is proven restart-free later, extending
CONTROL_FAST_PATH_KEYSis aone-line change with its own evidence trail, per the closed-allowlist design.
Files changed
rigforge.sh:CONTROL_FAST_PATH_KEYS,_control_fast_path_eligible(),_control_do_apply_fast()— new,inserted after
_control_do_apply().control_apply()— dispatches on_control_fast_path_eligibleinstead of unconditionallycalling
_control_do_apply; the failure/rollback branch is otherwise unchanged.tests/run.sh:_control_fast_path_eligibleclassification tests (== unit: _control_fast_path_eligible — closed allowlist classification (#381) ==)._control_do_apply_fastintegration test against a realinstall_watchdogrender (== unit: _control_do_apply_fast — reuses install_watchdog, skips apply()/xmrig restart (#381) ==).ca_exec's harness (== unit: control_apply orchestration + rollback (#236) ==) extended withfull-apply-called/fast-apply-calledmarker files and a_control_do_apply_faststub, plus newcontrol_apply()dispatch/fallback assertions.caf_exec/caf_runharness with a statefulsystemctlstub (== unit: control_apply + REAL _control_do_apply_fast — run-state criterion, not is-active alone (#381 security review) ==) — runscontrol_applyend to end with the real_control_do_apply_fastbody (notstubbed), since the fix under review is inside that function.
docs/adr/0001-writable-worker-config-control-path.md— new D12 decision record, updated tostate the run-state criterion (not the original is-active-alone claim).
docs/operations.md,docs/pithead-integration.md,docs/configuration.md— note the fast pathwhere they already describe
/apply's behavior and cost.CHANGELOG.md—[Unreleased]entry.Tests — what each one catches
All in
tests/run.sh, dependency-free (bash tests/run.sh/make test):_control_fast_path_eligible(pure classification):watchdog_interval_min/max_temp_calone → eligible; both together → eligiblereturn 0/return 1swapped)pools,DONATION,autotune,watchdogalone → not eligibleDONATION,max_temp_c(mixed) → not eligiblesome_future_keyalone → not eligible"?"→ not eligibleCONTROL_FAST_PATH_KEYSextracted from the script == exactly{max_temp_c, watchdog_interval_min}CONTROL_WRITABLE_KEYSentry (dead code)_control_do_apply_fast(against a realinstall_watchdogrender):apply()marker file never writtenOnUnitActiveSecreflects the new intervalinstall_watchdognot actually being invoked, or invoked with stale valuesconfig_meta.source==control,last_change_id== the test's cidcontrol_apply()+ the REAL_control_do_apply_fast, against a statefulsystemctlstub thatanswers the two
is-activecalls differently (the generic always-succeeds stub used everywhere elsecan't exercise this — that gap is exactly what the review flagged):
applied, new value lands inconfig.json, no rollback/restart attemptedrolled_back, config restored, rollback re-apply invokedcontrol_apply()orchestration (extends the existing#236rollback harness withfull-apply-called/fast-apply-calledmarkers):watchdog_interval_min-only /max_temp_c-only / both-together → fast marker set, full marker absent, statusappliedDONATION,max_temp_c(mixed) → full marker set, fast marker absentDONATION-only → full marker set, fast marker absentCA_FAST_APPLY_OK=0) → statusrolled_back, config restored, rollback re-apply uses the full pathappliedregardless of the fast apply's return codeWhat was run
The 1 failure (
msr-apply: missing wrmsr warns, never fails the unit (#140)) is a pre-existing,unrelated environment flake: this dev box has a real
/usr/sbin/wrmsrbinary that leaks into thattest's PATH stub, so it emits "Permission denied" instead of the expected "not found" — already
tracked with its own fix (
test: pin the missing-wrmsr case's PATH so the host's real wrmsr cannot leak in). Every assertion this PR adds passed; re-ran withgrep -c "(#381)"isolated to confirmnone were red.
Not run:
make test-e2e(container e2e),make e2e-real/make e2e-pithead(real-rig/bench gates) —this change is unit/repo-level only, per the task scope; no real miner or bench hardware was touched.
Follow-up filed separately (not in this PR)
The pre-existing hazard described under "Security review fix" above (
_apply_runtime's unconditionalrestart on the full apply/rollback path, independent of #381) is drafted as its own issue and will be
filed after this PR — deliberately out of scope here to keep this change to the fast path it was
asked to add.
Closes #381.