Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ All notable changes to RigForge are documented here. The format is based on

## [Unreleased]

### Added

- **Control API: a restart-free fast path for `watchdog_interval_min` and `max_temp_c` (#381, from
#344 item 1).** `control-apply` used to re-run the entire `apply` pipeline for every accepted
change — regenerate XMRig's config, re-render its unit, restart the service, then poll for a live
pool connection — even for a change that never touches XMRig at all. A change whose keys are
*only* `watchdog_interval_min` and/or `max_temp_c` now reconciles just the watchdog timer and
leaves XMRig running, closing the ~62s gap the original walkthrough measured for a single-key
change. The allowlist is closed and checked as a subset match (a key not on it, including any
future addition to the control-writable set, still takes the full path), and a fast-path failure
falls back to the same full-pipeline rollback a failed restart already uses.

## [1.15.2] - 2026-08-21

### Added
Expand Down
6 changes: 6 additions & 0 deletions docs/adr/0001-writable-worker-config-control-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,12 @@ D7 stamps `source: "control"` on a control apply. #254 extends the provenance to

The accepted→poll contract (D2) has a race: a concurrent change between a caller's `POST` and its poll makes the no-arg `GET /status` report the *newer* change, so the caller can't confirm its own. #255 adds `GET /status?change_id=<16hex>` returning that change's recorded outcome (or `404`) — the applier already writes each outcome, so it additionally indexes them under `changes/<change_id>.json` (last ~20; the id is server-generated 16-hex and re-validated before it becomes a path component). The no-arg form stays most-recent for compatibility; auth is unchanged. Chosen over a `/changes` ring-buffer endpoint (issue Option B) as the smaller, direct change.

### D12. A restart-free fast path for a closed subset of the allowlist (#381, from #344 item 1)

D6's `apply` re-run is correct as a baseline but expensive: a live walkthrough measured a single `watchdog_interval_min` change taking ~62s round-trip through `POST /apply`, because the applier re-runs the *entire* `apply` pipeline — regenerate XMRig's config, re-render its unit, restart the service, then poll for a live pool connection — for every change, even one that never touches XMRig at all.

The #344 discussion set one constraint before this could land: the fast path must not fork a second apply implementation that can drift from the real one. Decision: a closed allowlist (`watchdog_interval_min`, `max_temp_c`), checked as a **subset** match — every changed key must be on it, never a "not otherwise restart-requiring" complement — so a future addition to the D3 allowlist that nobody has re-proven restart-free here takes the full path by construction. Both current members are proven restart-free from the applier's own code, not asserted: `watchdog_interval_min` bakes into *only* `rigforge-watchdog.timer`'s cadence, and `max_temp_c` is never rendered into a unit at all — the watchdog verb re-reads `config.json` on every scheduled run. Neither reaches XMRig's generated config or its unit template. The fast path reuses `install_watchdog` — the same call the full `apply` pipeline already makes on every run — rather than re-implementing unit rendering, so the two paths cannot drift on what "restart-free" renders; it still stamps `config_meta` provenance (D10) the same way `apply` does, so a consumer cannot tell which path served a change from the feed alone. Success is a *run-state* comparison, not a bare is-active snapshot: the applier records whether the miner service was active *before* the fast apply and again *after*, and only a transition from active to inactive counts as failure. A rig can be legitimately stopped when a restart-free change lands — a watchdog thermal hold (D8/#257), or an operator's manual stop — and since the fast path never touches the XMRig unit or service, a stopped rig staying stopped (or even coming back on its own) is not this change's doing and must not be read as a fast-path failure; the new value still takes effect on the watchdog's next scheduled tick. An adversarial review of the first version of this decision found it gated on is-active alone, which would have discarded the operator's change and force-restarted a rig that was deliberately offline — exactly the thermal-hold-plus-`max_temp_c`-edit case this fast path exists for. A failure (active before, inactive after) falls through to the *same* full-pipeline rollback D6 already defines, never a bespoke recovery path. `pools`, `DONATION`, `autotune`, and the `watchdog` enable/disable flag stay on the full path: the first two are XMRig's own served config, and the latter two touch `install_*` behaviour this issue did not audit for restart-freedom.

## Alternatives considered

- **Write verbs on the sister API.** Rejected: violates the read-only invariant, and the `DynamicUser` read process cannot persist or apply.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ not read by `parse_config` today, so it isn't in the table below.)
| `api` | `"disabled"` | `"enabled"` serves the sister API: a second **read-only** port with XMRig's `/1/summary`+`/2/summary` passed through verbatim plus a namespaced `rigforge` object (tune state, RAPL watts, firmware/health probes, pinned versions), and `/health` + `/tune` endpoints. One tiny persistent stdlib server; a systemd timer refreshes its data every 15 s, so requests never touch the miner (see [operations › sister API](operations.md)). Gated by the same `ACCESS_TOKEN`; Linux-only. |
| `api_port` | `8081` | Sister API port (8080 is rejected — that's XMRig's own API). |
| `api_bind` | `"0.0.0.0"` | Sister API listen address. |
| `control` | `"disabled"` | `"enabled"` serves the **writable** control path (#236): a *separate* authenticated port that lets a Pithead stack apply config changes through RigForge, so `config.json` stays the source of truth (the producer for pithead Worker Inspect). **Fail-closed:** enabling it requires *both* `ACCESS_TOKEN` and `api_allow_from` — a writable API with no token or no pinned source is refused with a hard error. Only `pools`, `DONATION`, `autotune`, `watchdog`(+`watchdog_interval_min`), and `max_temp_c` are writable through it; anything else is rejected. The **remote** path additionally refuses to disable `watchdog` or to unset / out-of-band `max_temp_c` — a rig's thermal protection can only be *removed* by a local `rigforge.sh apply` on the box (#257). Each change is validated, the old config is snapshotted to `config-backups/` first, and a change that doesn't come back live is rolled back. The receiver holds no privilege and stages off the request path, so writes never touch mining. Linux-only. See [Operations › Control path](operations.md#writable-control-path-opt-in). |
| `control` | `"disabled"` | `"enabled"` serves the **writable** control path (#236): a *separate* authenticated port that lets a Pithead stack apply config changes through RigForge, so `config.json` stays the source of truth (the producer for pithead Worker Inspect). **Fail-closed:** enabling it requires *both* `ACCESS_TOKEN` and `api_allow_from` — a writable API with no token or no pinned source is refused with a hard error. Only `pools`, `DONATION`, `autotune`, `watchdog`(+`watchdog_interval_min`), and `max_temp_c` are writable through it; anything else is rejected. A change touching only `watchdog_interval_min` and/or `max_temp_c` applies without restarting XMRig (#381) — every other key restarts it. The **remote** path additionally refuses to disable `watchdog` or to unset / out-of-band `max_temp_c` — a rig's thermal protection can only be *removed* by a local `rigforge.sh apply` on the box (#257). Each change is validated, the old config is snapshotted to `config-backups/` first, and a change that doesn't come back live is rolled back. The receiver holds no privilege and stages off the request path, so writes never touch mining. Linux-only. See [Operations › Control path](operations.md#writable-control-path-opt-in). |
| `control_port` | `8082` | Control path port (rejects 8080 and the `api_port`). |
| `control_bind` | `"0.0.0.0"` | Control path listen address. Pair with `api_allow_from` (required) to pin who may write. |
| `miner_user` | `""` *(root)* | Run the miner as this dedicated non-root system user (created at setup, nologin). RigForge applies the CPU's MSR preset root-side before start; on families without a known preset the ~10-15% MSR boost is skipped — which is why this ships opt-in. Lowering privilege changes nothing else: HugePages come from the boot reservation, tune/doctor/apply all keep working. |
Expand Down
14 changes: 10 additions & 4 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -590,10 +590,16 @@ How a change flows:
2. A path-triggered root oneshot picks up the staged change, snapshots the current `config.json` to
`config-backups/config-<UTC-stamp>.json`, merges only the allowlisted keys, and re-validates the
result with the same rules `apply` uses. An invalid change is rejected and **nothing is written**.
3. A valid change is written durably (temp file, `fsync`, atomic rename) and applied through the
normal `apply` path. If the miner does not come back to a live hashrate, the snapshot is restored
and re-applied, and the outcome is recorded as `rolled_back` (or `failed`, if the rollback snapshot
itself could not be read back).
3. A valid change is written durably (temp file, `fsync`, atomic rename) and applied. Changing only
`watchdog_interval_min` and/or `max_temp_c` takes a **restart-free fast path** (#381): neither key
ever reaches XMRig's generated config or its unit, so only the watchdog timer is reconciled and
XMRig itself is left running — round-trip in about a second instead of the ~60s a full restart
costs on a big-page host. Any other key (alone or mixed with those two) applies through the normal
`apply` path, which restarts XMRig. Either way, if the miner does not come back to a live hashrate
(full path) or the miner service is not found still running (fast path — a signal something else
was already wrong, since neither key can cause that), the snapshot is restored and re-applied
through the full path, and the outcome is recorded as `rolled_back` (or `failed`, if the rollback
snapshot itself could not be read back).
4. `GET :8082/status` returns the last change's outcome (`applied` / `rejected` / `rolled_back` / `failed`, with
`source: "control"`, the changed keys, the backup path, and a `warnings[]` for any change that
touched thermal protection). Every response also carries a derived `age_seconds` next to its own
Expand Down
6 changes: 5 additions & 1 deletion docs/pithead-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,11 @@ apply config changes through RigForge — the RigForge-side producer for pithead
(pithead #185). It is deliberately independent of the read API: a `POST :8082/apply` of an
allowlisted change (`pools`, `DONATION`, `autotune`, `watchdog`, `watchdog_interval_min`,
`max_temp_c`) returns `202 Accepted`; RigForge validates, snapshots the old config, applies it, and
rolls back anything that doesn't come back live. The stack reads the new effective config back from
rolls back anything that doesn't come back live. Changing only `watchdog_interval_min` and/or
`max_temp_c` never restarts XMRig (#381) — those two are proven not to reach XMRig's config or unit,
so RigForge reconciles just the watchdog timer instead of running the full apply pipeline, landing in
about a second instead of the ~60s a restart costs. Any other key, alone or mixed with those two,
takes the full, XMRig-restarting path. The stack reads the new effective config back from
`:8081/2/summary` and polls `:8082/status` for the outcome. The write path is pinned to the stack
host by `api_allow_from` (mandatory) — the miner never accepts a config from anywhere else. Full
mechanics and the security model: [Operations › Writable control path](operations.md#writable-control-path-opt-in)
Expand Down
105 changes: 102 additions & 3 deletions rigforge.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4134,6 +4134,89 @@ _control_do_apply() {
_wait_miner_live "${CONTROL_LIVE_TRIES:-20}"
}

# #381 (from #344 item 1): a live walkthrough measured a single restart-free key
# (watchdog_interval_min) taking ~62s round-trip through POST /apply, because _control_do_apply
# above re-runs the ENTIRE apply() pipeline — regenerate xmrig's config, re-render its unit, restart
# the service, then wait out _wait_miner_live's pool-liveness retries — for every change, even one
# that never touches xmrig at all.
#
# CONTROL_FAST_PATH_KEYS is a CLOSED allowlist, checked as a SUBSET match (never a "not on the slow
# list" complement) by _control_fast_path_eligible below: a changed-keys set takes the fast path only
# when EVERY key in it is in this list, so an unrecognised key — including any future addition to
# CONTROL_WRITABLE_KEYS that nobody has re-proven restart-free here — falls through to the full,
# restart-safe path by construction. Evidence per key (grep rigforge.sh for both to re-check):
#
# watchdog_interval_min — bakes into ONLY rigforge-watchdog.timer's OnUnitActiveSec, rendered by
# install_watchdog. It never reaches generate_xmrig_config or the xmrig unit template, so xmrig
# itself has nothing to reload; install_watchdog (the SAME function the full path already calls
# unconditionally on every apply, see apply() above) is sufficient — it re-renders the timer and
# daemon-reloads it.
# max_temp_c — never rendered into ANY unit. install_watchdog's own comment records why: "only the
# cadence is baked into the units — the verb re-reads config.json every run, so an apply after a
# max_temp_c ... edit needs no unit rewrite". The watchdog verb picks it up on its next scheduled
# run; nothing beyond the config.json write _control_commit already did is needed here.
#
# pools/DONATION change xmrig's OWN generated config — restarting it is the only way xmrig serves the
# new values — and autotune/watchdog (the enable/disable flag, not the interval) govern install_*
# paths this issue has not audited for restart-freedom. All four stay on the full path. Never widen
# this list without the same kind of evidence trail (a grep proving the key never reaches
# generate_xmrig_config or a unit template); the closed-set-subset design means an unaudited key's
# safe default is already "restart", not "skip".
CONTROL_FAST_PATH_KEYS="watchdog_interval_min max_temp_c"

# True (rc 0) iff <keys-csv> is non-empty and every key in it is in CONTROL_FAST_PATH_KEYS. A single
# key outside the list — or an empty/malformed keys-csv (e.g. control_apply's "?" sentinel when jq
# couldn't read the staged file's keys) — fails CLOSED to "not eligible", so the caller takes the
# full apply() path. Subset match, never complement: this can only be talked into "eligible" by a key
# that is actually named on the allowlist above, so the allowlist growing is the only way this
# function's answer changes for a given input.
_control_fast_path_eligible() { # <keys-csv>
local keys="$1" k found=0
[ -n "$keys" ] || return 1
local IFS=','
for k in $keys; do
[ -n "$k" ] || continue
found=1
case " $CONTROL_FAST_PATH_KEYS " in
*" $k "*) ;;
*) return 1 ;;
esac
done
[ "$found" -eq 1 ]
}

# The restart-free counterpart to _control_do_apply (#381). Only reached when
# _control_fast_path_eligible says every changed key is provably restart-free (see
# CONTROL_FAST_PATH_KEYS above), so this DELIBERATELY skips apply()/_apply_runtime entirely: no
# generate_xmrig_config, no xmrig unit re-render, no `systemctl restart` of the miner, no
# _wait_miner_live retry loop, no _apply_pool_check. It reuses install_watchdog verbatim — the same
# call the full apply() makes unconditionally on every run — rather than re-implementing unit
# rendering, so the fast and full paths cannot drift on what "restart-free" actually renders. It
# still stamps provenance exactly like apply() does, via the same RIGFORGE_CONFIG_SOURCE /
# RIGFORGE_CONFIG_CHANGE_ID dynamic-scope contract, so config_meta on the read feed does not depend on
# which path served the change.
#
# Success is "the run-state did not DEGRADE", not "is currently active" (security review finding on
# the original version of this function, which gated on is-active alone): a rig can be LEGITIMATELY
# stopped before this change lands — a watchdog thermal hold, or an operator's manual stop — and by
# construction the fast path never touches the xmrig unit or service, so a stopped rig staying
# stopped (or even coming back up on its own) is correct, not a failure; the new value takes effect
# on its own schedule (the watchdog's next tick). Only a transition from active to inactive is a real
# regression worth the caller's rollback. No thermal-hold-marker special-casing needed: the
# before/after comparison covers every "was already down" case with less machinery.
_control_do_apply_fast() {
local was_active is_active
was_active=1
systemctl is-active --quiet "$SERVICE_NAME" 2>/dev/null && was_active=0
parse_config
install_watchdog >/dev/null 2>&1 || true
_stamp_config_meta "${RIGFORGE_CONFIG_SOURCE:-local}" "${RIGFORGE_CONFIG_CHANGE_ID:-}"
is_active=1
systemctl is-active --quiet "$SERVICE_NAME" 2>/dev/null && is_active=0
[ "$was_active" -eq 1 ] && return 0
[ "$is_active" -eq 0 ]
}

# Record a status record for the receiver's GET /status (mode 644 so the DynamicUser server reads it
# back) — a terminal outcome, or control_upgrade's non-terminal `started` marker (#320).
_control_status() { # <status-file> <status> <cid> <keys-csv> <reason> <backup>
Expand Down Expand Up @@ -4198,13 +4281,29 @@ control_apply() {
fi
backup="${result#committed }"
_reown_config_backups "$backups"
log "control-apply: committed change $cid (keys: $change_keys); applying..."
# #254: attribute this (and the rollback re-apply) to the control path with its change_id — the
# nested apply()'s _stamp_config_meta reads these via dynamic scope.
local RIGFORGE_CONFIG_SOURCE=control RIGFORGE_CONFIG_CHANGE_ID="$cid"
if _control_do_apply; then
# #381: dispatch on the closed fast-path allowlist. Only the CLASSIFICATION differs between the
# two branches below — a success writes the same "applied" status either way, and a failure of
# EITHER path falls through to the same full-pipeline rollback, so a wrong "eligible" verdict (or
# the fast path failing for an unrelated reason) still ends up restart-safe, never silently stuck.
local fast=0 apply_ok=0
if _control_fast_path_eligible "$change_keys"; then
fast=1
log "control-apply: committed change $cid (keys: $change_keys); applying (fast path — xmrig untouched)..."
_control_do_apply_fast && apply_ok=1
else
log "control-apply: committed change $cid (keys: $change_keys); applying..."
_control_do_apply && apply_ok=1
fi
if [ "$apply_ok" -eq 1 ]; then
_control_status "$status" applied "$cid" "$change_keys" "" "$backup"
log "control-apply: change $cid applied."
if [ "$fast" -eq 1 ]; then
log "control-apply: change $cid applied (fast path)."
else
log "control-apply: change $cid applied."
fi
else
warn "control-apply: change $cid did not come back live — rolling back to $backup."
# #276: the backup must be readable to restore it — guard the cp explicitly (not just -e/ERR)
Expand Down
Loading