diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ec48a3..b4ee78b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ All notable changes to RigForge are documented here. The format is based on ## [Unreleased] +### Fixed + +- **A failed watchdog re-render no longer reports the change as applied (#395).** `install_watchdog` + ended on `systemctl enable ... || true`, so it returned success whatever had happened above it, and + both apply paths called it under `|| true` — which also suppresses `set -e` for the whole call, so + a unit write that could not land aborted nothing and surfaced nowhere. A control `/apply` changing + `watchdog_interval_min` therefore recorded `applied` while the rig kept its old cadence, and nothing + detected the divergence. The function now returns an honest status, `apply` and the restart-free + fast path both act on it, and the outcome is recorded as `rolled_back` with a `reason` naming the + watchdog instead of blaming a hashrate that never dropped. + ## [1.16.0] - 2026-08-23 ### Added diff --git a/docs/operations.md b/docs/operations.md index 734156c..4d668d0 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -54,7 +54,7 @@ Not every verb needs root — the design, in four lines: |---|---| | `setup` *(default)* | Provision the worker: dependencies, build, hardware + kernel tuning, and the service. Idempotent and safe to re-run; skips the recompile when the pinned XMRig is already built. | | `upgrade` | Rebuild and restart only if the pinned XMRig version/commit changed. A no-op when you're already on the pinned build. If periodic autotune is enabled, it also re-tunes the new build (the fastest knobs can shift between versions). `--check` just reports whether a newer RigForge release exists (on-demand GitHub query, always exits 0). | -| `apply` | Re-read `config.json`, regenerate the live XMRig config, and restart, without recompiling. The fast path after editing `config.json`. On Linux it also reconciles the periodic-autotune timer with config (so changing the `autotune` target takes effect) and reports it (efficiency / performance / disabled), then asks the running miner whether a live pool connection came up — and warns when none did (a bad `pools[0].url` otherwise mines nothing while looking applied, #343). Warn only, never a failure: the pool may be legitimately down at apply time. | +| `apply` | Re-read `config.json`, regenerate the live XMRig config, and restart, without recompiling. The fast path after editing `config.json`. On Linux it also reconciles the periodic-autotune timer with config (so changing the `autotune` target takes effect) and reports it (efficiency / performance / disabled), then asks the running miner whether a live pool connection came up — and warns when none did (a bad `pools[0].url` otherwise mines nothing while looking applied, #343). Warn only, never a failure: the pool may be legitimately down at apply time. `apply` does exit non-zero if the watchdog units could not be re-rendered (#395) — that one is a real failure, because the cadence the rig is running would otherwise diverge from `config.json` silently. | | `uninstall` | Remove the service and revert all system changes (fstab, limits, modules, GRUB) and the worker build/logs. Leaves `config.json`. Prompts first; add `--yes` to skip. | | `doctor` | Read-only health check (run with `sudo` for the deepest checks). Critical findings (counted as issues): the service is active, the miner has a live pool connection (read from the miner's own local API — a running miner with no connection is not mining, #343; with the service stopped, one TCP dial of `pools[0]` stands in), HugePages are reserved, kernel lockdown isn't blocking MSR writes (read from `/sys/kernel/security/lockdown` — active lockdown means the MSR mod cannot apply, and `msr.allow_writes=on` can't override it), the `msr` module is loaded, and the MSR mod actually applied, confirmed from XMRig's log and, as root, an `rdmsr` register read-back (see [MSR mod verification](#msr-mod-verification)). Advisory findings (hints, not failures): CPU governor, 1 GB HugePages, HugePages 100%-backed (from the XMRig log), hashrate-capping hardware RigForge can't fix but you can (single-channel or slow RAM via `dmidecode`, and a power/boost-capped CPU clock), and BIOS/firmware recommendations (board/BIOS context, plus enable XMP/EXPO/DOCP or SMT when they're off; manual BIOS changes RigForge can't make from the OS). Prints an actionable hint for anything off. Also binary tamper evidence (#141): the on-disk `xmrig` is compared against the SHA-256 recorded at compile time — a deliberate rebuild refreshes the record, anything else warns and counts as an issue. When the [writable control path](#writable-control-path-opt-in) is enabled, it's also a counted issue: `doctor` checks `rigforge-control` is active and its `/status` endpoint answers (#278), staying silent when control is disabled. Exits non-zero when critical issues are found (cron-friendly, matching Pithead's `status`). | | `bench` | Run a one-off `xmrig --bench` and report the hashrate (a quick perf/health check; set `BENCH=10M` for a longer run). | @@ -599,7 +599,11 @@ How a change flows: (full path) or the miner service is not found still running (fast path — a signal something else was already wrong, since neither key can cause that), the snapshot is restored and re-applied through the full path, and the outcome is recorded as `rolled_back` (or `failed`, if the rollback - snapshot itself could not be read back). + snapshot itself could not be read back). A watchdog timer that could not be re-rendered — the unit + files are unwritable, or `daemon-reload` fails — rolls the change back the same way (#395). The + fast path's whole effect *is* that re-render, so recording the change as applied would leave the + rig on a cadence the operator never chose, with nothing to detect it. The recorded `reason` names + the watchdog rather than the hashrate, which in that case never dropped. 4. `GET :8082/status` returns the last change's outcome (`applied` / `rejected` / `rolled_back` / `failed`, with `source: "control"`, the changed keys, the backup path, and a `warnings[]` for any change that touched thermal protection). Every response also carries a derived `age_seconds` next to its own diff --git a/rigforge.sh b/rigforge.sh index e6de5cd..b012907 100755 --- a/rigforge.sh +++ b/rigforge.sh @@ -1272,26 +1272,39 @@ install_autotune() { install_watchdog() { [ "$OS_TYPE" == "Linux" ] || return 0 local svc="$SYSTEMD_DIR/rigforge-watchdog.service" tmr="$SYSTEMD_DIR/rigforge-watchdog.timer" + # #395: this function used to end on `systemctl enable ... || true`, so it returned 0 no matter + # what happened above it — and its callers run it under `|| true`, which suppresses `set -e` for + # the whole dynamic extent of the call. A failed unit write therefore aborted nothing, surfaced + # nowhere, and left the caller free to record the change as applied. Each step that can GENUINELY + # fail now records into rc and the function returns it, so a caller that looks gets the truth and + # a caller that does not is no worse off than before. The `|| true` on enable/disable of a + # possibly-absent timer stays deliberate: those are tolerated, the unit writes are not. + local rc=0 if [ "${WATCHDOG_MODE:-disabled}" = "disabled" ]; then if [ -f "$tmr" ]; then sudo systemctl disable ${ENABLE_RUNTIME:+"$ENABLE_RUNTIME"} --now rigforge-watchdog.timer 2>/dev/null || true - sudo rm -f "$svc" "$tmr" + sudo rm -f "$svc" "$tmr" || rc=1 sudo systemctl daemon-reload 2>/dev/null || true - log "Miner watchdog disabled." + if [ "$rc" -eq 0 ]; then + log "Miner watchdog disabled." + else + warn "Could not remove the watchdog units ($svc, $tmr) — the old timer may still be running." + fi fi - return 0 + return "$rc" fi log "Enabling the miner watchdog: a health check every ${WATCHDOG_INTERVAL_MIN:-5} min${MAX_TEMP_C:+, thermal cutoff ${MAX_TEMP_C}°C}..." # Only the cadence is baked into the units — the verb re-reads config.json every run, so an # `apply` after a max_temp_c or ACCESS_TOKEN edit needs no unit rewrite (and no token on disk). SERVICE_NAME="$SERVICE_NAME" RIGFORGE_OPERATOR="$REAL_USER" SCRIPT_DIR="$SCRIPT_DIR" \ envsubst '$SERVICE_NAME $RIGFORGE_OPERATOR $SCRIPT_DIR' \ - <"$SCRIPT_DIR/systemd/rigforge-watchdog.service.template" | sudo tee "$svc" >/dev/null + <"$SCRIPT_DIR/systemd/rigforge-watchdog.service.template" | sudo tee "$svc" >/dev/null || rc=1 WATCHDOG_INTERVAL_MIN="${WATCHDOG_INTERVAL_MIN:-5}" \ envsubst '$WATCHDOG_INTERVAL_MIN' \ - <"$SCRIPT_DIR/systemd/rigforge-watchdog.timer.template" | sudo tee "$tmr" >/dev/null - sudo systemctl daemon-reload + <"$SCRIPT_DIR/systemd/rigforge-watchdog.timer.template" | sudo tee "$tmr" >/dev/null || rc=1 + sudo systemctl daemon-reload || rc=1 sudo systemctl enable ${ENABLE_RUNTIME:+"$ENABLE_RUNTIME"} --now rigforge-watchdog.timer 2>/dev/null || true + return "$rc" } # Sister API (#99/#164, xmrig-model): one tiny persistent python3-stdlib server ships pre-computed @@ -4029,9 +4042,15 @@ apply() { sudo systemctl daemon-reload 2>/dev/null || true fi _apply_runtime + local _wd_rc=0 if [ "$OS_TYPE" = Linux ]; then install_autotune >/dev/null 2>&1 || true - install_watchdog >/dev/null 2>&1 || true + # #395: of the install_* steps here, a failed watchdog re-render is the one that leaves the + # RUNNING cadence silently diverged from the config we are about to stamp as in effect, so it + # is captured instead of discarded. The remaining steps still run — abandoning the reconcile + # would strand more units than the one that failed — but apply() returns non-zero at the end, + # which is what lets control_apply reach its rollback branch instead of recording "applied". + install_watchdog >/dev/null 2>&1 || _wd_rc=1 install_api >/dev/null 2>&1 || true install_control >/dev/null 2>&1 || true install_api_firewall || true @@ -4043,6 +4062,11 @@ apply() { # a re-apply of the same config, never bump the revision). _stamp_config_meta "${RIGFORGE_CONFIG_SOURCE:-local}" "${RIGFORGE_CONFIG_CHANGE_ID:-}" _apply_pool_check + if [ "$_wd_rc" -ne 0 ]; then + RIGFORGE_APPLY_FAIL_REASON="could not re-render the watchdog units" + warn "The miner watchdog units could NOT be re-rendered ($SYSTEMD_DIR/rigforge-watchdog.{service,timer}) — the watchdog is still on its PREVIOUS schedule, whatever config.json now says. Check free disk space and sudo rights, then re-run 'sudo $0 apply'." + fi + return "$_wd_rc" } # #343: apply's honesty check. "Applied config and restarted" used to be the last word even when the @@ -4167,11 +4191,20 @@ _reown_config_backups() { # } # The apply + liveness check control-apply gates its rollback on. Split out so tests can stub it. +# #395: apply's exit status was discarded here — the liveness wait ran regardless and its verdict +# became the whole answer, so an apply that failed for a reason the miner's hashrate cannot show +# (a watchdog unit that would not write) still reported success. Gate on it: a failed apply is a +# failed apply, and control_apply's rollback branch is the designed response to one. _control_do_apply() { - apply >/dev/null 2>&1 + apply >/dev/null 2>&1 || return 1 _wait_miner_live "${CONTROL_LIVE_TRIES:-20}" } +# #395: the specific cause of the most recent apply failure, when the failing path knows one the +# liveness wait could not have observed. Set by that path, read and cleared by control_apply, which +# falls back to its existing wording when this is empty — so no pre-existing reason string changes. +RIGFORGE_APPLY_FAIL_REASON="" + # #381 (from #344 item 1): a live walkthrough measured a single restart-free key # (watchdog_interval_min) taking ~62s round-trip through POST /apply, because _control_do_apply # above re-runs the ENTIRE apply() pipeline — regenerate xmrig's config, re-render its unit, restart @@ -4247,7 +4280,15 @@ _control_do_apply_fast() { was_active=1 systemctl is-active --quiet "$SERVICE_NAME" 2>/dev/null && was_active=0 parse_config - install_watchdog >/dev/null 2>&1 || true + # #395: install_watchdog IS this path's entire effect — both fast-path keys reach the rig only + # through it. Swallowing its failure recorded "applied" for a change that never landed, which is + # the precise lie the status contract exists to prevent. Bail before stamping provenance: a + # change that did not take effect must not be stamped as the config in force. The caller then + # routes to the same full-pipeline rollback the comment above already promises for this case. + if ! install_watchdog >/dev/null 2>&1; then + RIGFORGE_APPLY_FAIL_REASON="could not re-render the watchdog units" + return 1 + fi _stamp_config_meta "${RIGFORGE_CONFIG_SOURCE:-local}" "${RIGFORGE_CONFIG_CHANGE_ID:-}" is_active=1 systemctl is-active --quiet "$SERVICE_NAME" 2>/dev/null && is_active=0 @@ -4326,7 +4367,11 @@ control_apply() { # two branches below — a success writes the same "applied" status either way, and a failure of # EITHER path falls through to the same full-pipeline rollback, so a wrong "eligible" verdict (or # the fast path failing for an unrelated reason) still ends up restart-safe, never silently stuck. - local fast=0 apply_ok=0 + local fast=0 apply_ok=0 fail_reason="" + # #395: clear before the attempt so a stale cause from an earlier change can never be attributed + # to this one, and snapshot it straight after so the ROLLBACK's own apply cannot overwrite the + # reason the change failed in the first place. + RIGFORGE_APPLY_FAIL_REASON="" if _control_fast_path_eligible "$change_keys"; then fast=1 log "control-apply: committed change $cid (keys: $change_keys); applying (fast path — xmrig untouched)..." @@ -4335,6 +4380,7 @@ control_apply() { log "control-apply: committed change $cid (keys: $change_keys); applying..." _control_do_apply && apply_ok=1 fi + fail_reason="$RIGFORGE_APPLY_FAIL_REASON" if [ "$apply_ok" -eq 1 ]; then _control_status "$status" applied "$cid" "$change_keys" "" "$backup" if [ "$fast" -eq 1 ]; then @@ -4343,7 +4389,7 @@ control_apply() { log "control-apply: change $cid applied." fi else - warn "control-apply: change $cid did not come back live — rolling back to $backup." + warn "control-apply: change $cid ${fail_reason:-did not come back live} — rolling back to $backup." # #276: the backup must be readable to restore it — guard the cp explicitly (not just -e/ERR) # so an unreadable backup still writes a terminal status instead of ERR-trapping the oneshot # out silently, which would leave the receiver serving the stale previous outcome forever. @@ -4353,10 +4399,18 @@ control_apply() { fi # #276: distinguish "rolled back and live" from "rolled back, rig still down" — both restore # config.json, but only one leaves the miner hashing; the reason string pins which happened. + local why="${fail_reason:-miner did not return to a live hashrate}" if _control_do_apply; then - _control_status "$status" rolled_back "$cid" "$change_keys" "miner did not return to a live hashrate; rolled back and live" "$backup" + _control_status "$status" rolled_back "$cid" "$change_keys" "$why; rolled back and live" "$backup" + elif [ -n "$fail_reason" ]; then + # #395: the re-apply failed for the SAME reason the change did (units that still will not + # write), so the liveness wording below would assert something nobody checked — a failed + # apply short-circuits before _wait_miner_live runs, and the miner may be hashing fine + # throughout. Say what is known instead. The wording below is untouched for every + # pre-existing case, where the liveness wait genuinely is what returned the verdict. + _control_status "$status" rolled_back "$cid" "$change_keys" "$why; config rolled back, but the re-apply hit the same failure" "$backup" else - _control_status "$status" rolled_back "$cid" "$change_keys" "miner did not return to a live hashrate; rollback re-apply also failed to restore liveness" "$backup" + _control_status "$status" rolled_back "$cid" "$change_keys" "$why; rollback re-apply also failed to restore liveness" "$backup" fi fi return 0 diff --git a/tests/run.sh b/tests/run.sh index 4d0de99..0d34064 100644 --- a/tests/run.sh +++ b/tests/run.sh @@ -1934,6 +1934,70 @@ out="$(run_install_watchdog disabled)" assert_eq "watchdog disable removes the .timer" "$([ -f "$WDI/systemd/rigforge-watchdog.timer" ] && echo y || echo n)" "n" assert_eq "watchdog disable removes the .service" "$([ -f "$WDI/systemd/rigforge-watchdog.service" ] && echo y || echo n)" "n" +# #395: install_watchdog used to end on `systemctl enable ... || true`, so it returned 0 whatever had +# happened above it — and both apply-path callers run it under `|| true`, which suppresses `set -e` +# for the whole dynamic extent of the call, so an earlier failure neither aborted nor showed. A +# caller therefore could not distinguish "watchdog re-rendered" from "watchdog untouched". These pin +# the honest return value at the source, which is what every other #395 assertion below builds on. +# Mutation each case catches: turning that step's `|| rc=1` back into a bare call, or dropping the +# closing `return "$rc"` — either restores always-0 and reddens the rc assertion here. +echo "== unit: install_watchdog returns non-zero when it could not render (#395) ==" +WDF="$(mktemp -d "$SANDBOX/wdf.XXXXXX")" +mkdir -p "$WDF/systemd" "$WDF/bin" +cp "$ROOT/systemd/rigforge-watchdog.service.template" "$ROOT/systemd/rigforge-watchdog.timer.template" "$WDF/systemd/" +wdf_stub() { # + cat >"$WDF/bin/systemctl" <"$WDF/bin/rm" <"$WDF/bin/tee" <<'TEOF' +#!/usr/bin/env bash +if [ -n "${WDF_TEE_FAIL:-}" ] && [ "${*/$WDF_TEE_FAIL/}" != "$*" ]; then + cat >/dev/null + exit 1 +fi +exec /usr/bin/tee "$@" +TEOF + chmod +x "$WDF/bin/systemctl" "$WDF/bin/rm" "$WDF/bin/tee" +} +wdf_run() { # -> "rc=" + wdf_stub "$3" "$4" + ( + source "$SCRIPT" + OS_TYPE=Linux + SCRIPT_DIR="$WDF" + SYSTEMD_DIR="$1" + REAL_USER=rfop + SERVICE_NAME=xmrig + WATCHDOG_MODE="$2" + WATCHDOG_INTERVAL_MIN=7 + set +e + PATH="$WDF/bin:$STUBS:$PATH" install_watchdog >/dev/null 2>&1 + echo "rc=$?" + ) +} +# The healthy case first — without it the failure assertions below would also pass on a function that +# simply always returns 1, which proves nothing about honesty. +assert_eq "install_watchdog returns 0 when every step succeeds (#395)" "$(wdf_run "$WDF/systemd" enabled 0 0)" "rc=0" +assert_eq "a unit write that cannot land -> non-zero (#395)" "$(wdf_run "$WDF/absent/systemd" enabled 0 0)" "rc=1" +assert_eq "the .service write alone failing -> non-zero (#395)" "$(WDF_TEE_FAIL=rigforge-watchdog.service wdf_run "$WDF/systemd" enabled 0 0)" "rc=1" +assert_eq "the .timer write alone failing -> non-zero (#395)" "$(WDF_TEE_FAIL=rigforge-watchdog.timer wdf_run "$WDF/systemd" enabled 0 0)" "rc=1" +assert_eq "a failed daemon-reload -> non-zero (#395)" "$(wdf_run "$WDF/systemd" enabled 1 0)" "rc=1" +# The disabled branch is the same lie in the other direction: a timer that could not be REMOVED is +# still firing on the old cadence while config.json says the watchdog is off. +assert_eq "disable returns 0 when the units are actually removed (#395)" "$(wdf_run "$WDF/systemd" disabled 0 0)" "rc=0" +wdf_run "$WDF/systemd" enabled 0 0 >/dev/null # re-render so there is something to fail to remove +assert_eq "a unit that cannot be removed -> non-zero (#395)" "$(wdf_run "$WDF/systemd" disabled 0 1)" "rc=1" + # #139: the watchdog config keys. Typo hard-errors (a recovery mechanism must not be silently # disabled); the interval and cutoff validate as bounded integers; max_temp_c empty = cutoff off. echo "== unit: parse_config — watchdog keys (#139) ==" @@ -7236,6 +7300,41 @@ assert_contains "install_watchdog re-renders the timer with the NEW interval (#3 assert_eq "config_meta stamped source=control, parity with apply()'s own _stamp_config_meta call (#381)" "$(jq -r .source "$FPA/meta.json" 2>/dev/null)" "control" assert_eq "config_meta records the change_id, same parity (#381)" "$(jq -r .last_change_id "$FPA/meta.json" 2>/dev/null)" "fedcba9876543210" +# #395: install_watchdog IS this path's entire effect — both fast-path keys reach the rig only +# through it — so swallowing its failure recorded a change as applied that never landed. Two things +# must hold when it fails: a non-zero return (so the caller can roll back), and NO provenance stamp +# (a change that did not take effect must not be recorded as the config in force). +# Mutation this catches: restoring `install_watchdog >/dev/null 2>&1 || true` here — the mutant +# returns rc=0 AND stamps meta.json, reddening both assertions. +fpa_run_unwritable() { # -> "rc="; templates still readable, units unwritable + rm -f "$FPA/apply-called" "$FPA/meta.json" + ( + source "$SCRIPT" + OS_TYPE=Linux + SCRIPT_DIR="$FPA" + SYSTEMD_DIR="$FPA/absent/systemd" + REAL_USER=rfop + SERVICE_NAME=xmrig + CONFIG_JSON="$FPA/config.json" + CONFIG_META_FILE="$FPA/meta.json" + WATCHDOG_MODE=enabled + WATCHDOG_INTERVAL_MIN=9 + parse_config() { :; } + apply() { + echo called >"$FPA/apply-called" 2>/dev/null + return 0 + } + RIGFORGE_CONFIG_SOURCE=control + RIGFORGE_CONFIG_CHANGE_ID=fedcba9876543210 + set +e + PATH="$STUBS:$PATH" _control_do_apply_fast + echo "rc=$?" + ) +} +out="$(fpa_run_unwritable)" +assert_contains "_control_do_apply_fast fails when the watchdog units cannot be written (#395)" "$out" "rc=1" +assert_eq "a fast path that failed does NOT stamp the change as in force (#395)" "$([ -f "$FPA/meta.json" ] && echo stamped || echo not-stamped)" "not-stamped" + echo "== unit: control_apply + REAL _control_do_apply_fast — run-state criterion, not is-active alone (#381 security review) ==" # The generic systemctl stub always exits 0, so the fpa_run tests above only ever exercise the # active-before/active-after case. A rig can be LEGITIMATELY stopped when a restart-free change @@ -7271,12 +7370,12 @@ caf_exec() { # failure would cause. apply() { echo called >"$CAF/full-apply-called" 2>/dev/null || true - return 0 + return "${CAF_APPLY_OK:-0}" } _wait_miner_live() { return 0; } OS_TYPE=Linux SCRIPT_DIR="$CAF" - SYSTEMD_DIR="$CAF/systemd" + SYSTEMD_DIR="${CAF_SYSTEMD_DIR:-$CAF/systemd}" CONFIG_JSON="$CAF/config.json" REAL_USER=rfop SERVICE_NAME=xmrig @@ -7313,6 +7412,100 @@ assert_eq "active-before/inactive-after -> status rolled_back (#381)" "$(cfst st assert_eq "active-before/inactive-after -> config restored, max_temp_c unset again (#381)" "$(jq -r .max_temp_c "$CAF/config.json")" "null" assert_eq "active-before/inactive-after -> rollback re-apply invoked (#381)" "$([ -f "$CAF/full-apply-called" ] && echo called || echo not-called)" "called" +# (c) #395, and the reason this issue exists: the miner is up before AND after (rc 0, 0), so every +# liveness-shaped check this path has says success — but the watchdog units could not be written, so +# the cadence the operator just accepted is NOT the cadence the rig is running. The old code recorded +# "applied" here. This is the end-to-end assertion that the status record stopped lying; it runs the +# REAL _control_do_apply_fast and the REAL install_watchdog, with only the rollback leg stubbed. +# Mutation this catches: any single revert in the #395 chain — install_watchdog returning 0 again, or +# the fast path swallowing it — puts status back to "applied" and reddens the first assertion. +caf_run '{"max_temp_c":90}' 0 0 +assert_eq "healthy watchdog render, miner up throughout -> applied (#395 control)" "$(cfst status)" "applied" +CAF_SYSTEMD_DIR="$CAF/absent/systemd" caf_run '{"max_temp_c":90}' 0 0 +assert_eq "watchdog units unwritable, miner never down -> rolled_back, NOT applied (#395)" "$(cfst status)" "rolled_back" +# The reason string must name the actual cause. Falling back to the liveness wording here would be a +# fresh lie of the same kind: the miner never left the pool. +assert_contains "the recorded reason names the watchdog, not a liveness failure (#395)" "$(cfst reason)" "watchdog" +assert_absent "the recorded reason does NOT blame the hashrate (#395)" "$(cfst reason)" "live hashrate" +assert_eq "watchdog-failed change -> config restored to pre-change (#395)" "$(jq -r .max_temp_c "$CAF/config.json")" "null" +assert_eq "watchdog-failed change -> the full restart-safe rollback ran (#395)" "$([ -f "$CAF/full-apply-called" ] && echo called || echo not-called)" "called" +# And when the ROLLBACK's own re-apply fails for that same reason, the record must not fall back to +# the liveness wording: a failed apply short-circuits before the liveness wait, so nobody checked it. +# Mutation this catches: dropping the `elif [ -n "$fail_reason" ]` arm — the mutant reports "failed +# to restore liveness" about a miner this test never took down. +CAF_SYSTEMD_DIR="$CAF/absent/systemd" CAF_APPLY_OK=1 caf_run '{"max_temp_c":90}' 0 0 +assert_eq "watchdog failure + failed rollback re-apply -> still rolled_back (#395)" "$(cfst status)" "rolled_back" +assert_contains "the reason names the watchdog cause (#395)" "$(cfst reason)" "watchdog" +assert_contains "the reason says the re-apply hit the same failure (#395)" "$(cfst reason)" "re-apply hit the same failure" +assert_absent "it does NOT claim a liveness failure nobody checked (#395)" "$(cfst reason)" "restore liveness" + +# #395: the two seams between install_watchdog and the status record. apply() swallowed the failure +# outright, and _control_do_apply then DISCARDED apply's exit status — the liveness wait's verdict +# became the whole answer, so an apply that failed for a reason the miner's hashrate cannot show +# still reported success. Both are covered here rather than through control_apply, because the +# control_apply harnesses stub apply() and so cannot see either seam. +echo "== unit: apply() and _control_do_apply propagate a watchdog render failure (#395) ==" +AWD="$(mktemp -d "$SANDBOX/awd.XXXXXX")" +mkdir -p "$AWD/systemd" +cp "$ROOT/systemd/rigforge-watchdog.service.template" "$ROOT/systemd/rigforge-watchdog.timer.template" "$AWD/systemd/" +awd_apply() { # -> "rc="; side effect: $AWD/reached-later-steps + rm -f "$AWD/reached-later-steps" + ( + source "$SCRIPT" + OS_TYPE=Linux + SCRIPT_DIR="$AWD" + SYSTEMD_DIR="$1" + REAL_USER=rfop + SERVICE_NAME=xmrig + WATCHDOG_MODE=enabled + WATCHDOG_INTERVAL_MIN=7 + # Everything apply() does EXCEPT install_watchdog is out of scope here and covered elsewhere, + # so it is stubbed away; install_watchdog stays REAL and fails (or not) on the dir passed in. + parse_config() { :; } + _apply_runtime() { :; } + install_autotune() { :; } + install_api() { echo later >"$AWD/reached-later-steps" 2>/dev/null || true; } + install_control() { :; } + install_api_firewall() { :; } + _autotune_apply_notice() { :; } + _stamp_config_meta() { :; } + _apply_pool_check() { :; } + set +e + PATH="$STUBS:$PATH" apply >/dev/null 2>&1 + echo "rc=$?" + ) +} +assert_eq "apply() returns 0 when the watchdog renders (#395)" "$(awd_apply "$AWD/systemd")" "rc=0" +assert_eq "apply() returns non-zero when the watchdog could not render (#395)" "$(awd_apply "$AWD/absent/systemd")" "rc=1" +# The reconcile must not be abandoned mid-way: stranding install_api/control/firewall as well would +# leave MORE units stale than the one that failed. This pins that the failure is recorded, not raised. +assert_eq "a failed watchdog render does not abort the rest of the reconcile (#395)" "$([ -f "$AWD/reached-later-steps" ] && echo ran || echo skipped)" "ran" + +# Mutation this catches: restoring `apply >/dev/null 2>&1` as its own statement (status discarded). +# That mutant runs the liveness wait anyway, so the marker appears and rc drops to 0. +cda_run() { # -> "rc="; side effect: $AWD/waited + rm -f "$AWD/waited" + ( + source "$SCRIPT" + # Capture the helper's argument BEFORE defining apply: inside apply's own body, "$1" would be + # apply's parameter, not this one — and unset under `set -u` it fails for the wrong reason, + # which made the short-circuit assertion below pass vacuously until this was caught. + _cda_rc="$1" + apply() { return "$_cda_rc"; } + _wait_miner_live() { + echo waited >"$AWD/waited" 2>/dev/null || true + return 0 + } + set +e + _control_do_apply + echo "rc=$?" + ) +} +assert_eq "_control_do_apply still waits for liveness when apply succeeds (#395)" "$(cda_run 0)" "rc=0" +assert_eq "apply succeeded -> the liveness wait really ran (#395)" "$([ -f "$AWD/waited" ] && echo waited || echo skipped)" "waited" +assert_eq "_control_do_apply fails when apply fails (#395)" "$(cda_run 1)" "rc=1" +assert_eq "a failed apply short-circuits the liveness wait (#395)" "$([ -f "$AWD/waited" ] && echo waited || echo skipped)" "skipped" + echo "== unit: control_apply orchestration + rollback (#236) ==" CA="$(mktemp -d "$SANDBOX/ca.XXXXXX")" ca_exec() {