Skip to content

refactor(runtime-host)!: make the Runtime Host State Root self-contained - #5429

Draft
Astro-Han wants to merge 22 commits into
apache:mainfrom
Astro-Han:refactor/runtime-root-self-contained
Draft

Astro-Han wants to merge 22 commits into
apache:mainfrom
Astro-Han:refactor/runtime-root-self-contained

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

Runtime Host coordination previously derived from the OS account home: the owner lock lived under the account data directory, the control namespace and Host credentials under the account cache, and the managed deployment record under the account data root. Accounts without a usable home (nobody, HOME=/tmp, a passwd home of /nonexistent) could not start a Runtime Host, and cleaning the account cache destroyed issued access credentials and disconnected locks from the root they protected.

This PR moves the whole coordination surface inside the physical root and bumps the root marker to schema version 2:

  • <root>/.maka-host/<rootId>.lock is the single owner lock; .maka-host/artifact-writer-bootstrap.lock is the single Artifact-writer lock.
  • .maka-host/state/data holds plugin state, plugin credentials, access credentials and composition; .maka-host/state/deployment/runtime-host-deployment.json is the sole managed deployment transaction record.
  • .maka-host/runtime holds disposable registration, startup diagnostics and one-time credential deliveries.
  • The account side keeps only a locator (root-location.json) plus service-manager bookkeeping; a missing locator is a discovery failure, not data loss.

Invariants preserved: one Host writer per physical root, rootId stays protocol identity, marker device/inode checks still reject copied directories, no clone support and no global duplicate-identity registry.

prepareRuntimeHostRoot is the single Host-side format admission point. Ordinary startup, activation, update, managed-service launch and deployment cleanup all pass through it, so a schema-1 root migrates automatically and an interrupted migration resumes forward from its durable fence without manual repair. All schema-1 knowledge is clustered in root-upgrade.ts behind inspectStorageRootFormat/withStorageRootUpgrade so the removal below is mechanical.

Refs #5320 (the privileged account-home workaround this replaces), #4712 (the account-cache control namespace this deletes), #1286.

Migration

Schema 2 is a destructive format change: older binaries cannot open a migrated root. Roots written by v0.1.x, cli-v0.1.0-beta.1, v0.2.0-incubating-rc1 or v0.2.0-dev builds migrate automatically on first admission under 0.2.x. Schema-1 support is limited to the 0.2.x line — starting with 0.3 the migration machinery is removed and unmigrated roots are rejected; a root that skips every 0.2.x release must be opened once by a 0.2.x build or restored from backup. Downgrading requires restoring a pre-upgrade copy of the root directory.

Verification

  • npm --workspace builds for storage, runtime, runtime-host, cli (and dependents) — clean.
  • Focused compiled suites via node --test --test-concurrency=4: storage root-authority (33 pass / 3 platform skips), runtime-host upgrade + two-client + managed + websocket batches, cli service-manager/lifecycle/activation/setup/update batches — all green, including the new interruption-fence matrix, the service-entry migration regression test and the deferred-cleanup-receipt test.
  • Docker nobody (passwd home /nonexistent) harness: root resolution, owner acquisition, control-directory preparation, reader contention and second-owner refusal all pass.
  • npm run format / npm run lint clean; python3 -m unittest harbor relay tests pass.
  • docs/windows-test-inventory.md regenerated; the upgrade suites are added to the windows-recovery lane, which has not run here (no local Windows).
  • Not run: full-repository test suite (out of scope per repo rules), Windows-native execution beyond the CI lane, release-to-release upgrade against published packages.

AI use

  • Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex authored the original candidate-branch commits (Generated-by trailers preserved); Devin rebased the branch onto main, added the namespace flattening and fixed-name bootstrap lock, the service/cleanup format-admission work, the relay workaround removal, the Windows lane update and the new regression tests (Generated-by: Devin trailers).

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above

Move owner and Artifact locks, durable Host data, and managed deployment authority into the physical State Root. Retain account-side root lookup only. Fence and recover the schema-1 migration before publishing schema 2; retire steady-state compatibility locks.

BREAKING CHANGE: schema-2 roots are rejected by older binaries. Migrate schema-1 roots while their recorded filesystem identity and legacy account state are available.

Generated-by: OpenAI Codex
Move legacy import out of root resolution into one fenced Host upgrade transaction. Persist takeover before copying, publish a complete state snapshot, and resume deployment activation through the existing lifecycle transaction.

Generated-by: OpenAI Codex
The control directory keeps no <rootId> leaf and the Artifact writer
bootstrap lock is a fixed root-local file: inside one physical root both
were redundant layers inherited from the shared account-side namespace.
The capability-facing controlDirectory result no longer exposes a
separate control root.

Generated-by: Devin
The managed service entry now runs the same recovery admission as
activation before serving, and plain service startup resolves the root
through prepareRuntimeHostRoot so a legacy root migrates instead of
failing launch. Deployment cleanup and retirement reaping check the
recorded root format first: a pre-migration root defers its receipt to
the post-migration staging pass instead of throwing inside the upgrade
fence. Management commands report root_requires_migration rather than a
target mismatch when the root still carries schema version 1.

Generated-by: Devin
The privileged setup step provisioned a missing passwd home only because
Storage coordinated through the system account home. Root-local state
makes an account home unnecessary for a fresh root, so the workaround
and its test are deleted.

Generated-by: Devin
Add the newly reachable sources to the path filter and run the upgrade
suites on the Windows runner.

Generated-by: Devin
@github-actions github-actions Bot added the effort/XXL Over 2500 readable lines label Sep 17, 2026
Astro-Han and others added 12 commits September 17, 2026 16:03
The migration plan now lives in upgrade-plan.json beside the root
authority instead of inside the 32KB marker, where a full target
deployment config could exceed the marker limit and wedge the root
permanently. A torn completion record restages instead of failing, a
deterministic staged-content failure reports its source paths and asks
for repair instead of replaying silently, and concurrent admissions wait
a bounded interval on root_migration_busy rather than failing outright.
Resumed plans are trusted only within the four legacy lock shapes, the
compatibility probe runs the recorded package through the current node
binary instead of trusting a stored nodePath, and lock-release failures
no longer mask the real migration error. connectOrSpawn passes the
injected managed authority to its legacy gate, and a stale legacy
locator no longer demotes a valid in-root authority record.

Generated-by: Devin
Managed recovery now settles the legacy lifecycle transaction through
its installed package before migrating the root format, so a v1 root
with an unfinished transition no longer deadlocks on the migration
order. serve --managed-deployment runs the same recovery admission as
activation, and the installed-update coordinator prepares the root
instead of resolving it directly so unmanaged legacy roots can update.
Expected-target verification reads root identity only, leaving
capability access to the operations that need it; management actions on
a pre-migration root report root_requires_migration or
root_migration_busy instead of an internal error. The deployment reaper
consults the live authority before the recorded cleanup path so a stale
receipt cannot block staging.

Generated-by: Devin
The released-CLI fixture writes its access credential to the owner's
hostDataDirectory so old-to-new qualification finds it in the schema 2
data directory, and the Windows diagnostics script derives the control
leaf from the installed package's marker schema instead of appending the
root id unconditionally. The package-validation path filter now covers
the migration and managed-deployment sources, and the Windows test
inventory classifies the POSIX-symlink fixtures as backend gaps.

Generated-by: Devin
…ption

A durable fence with an unreadable plan fell back to first-time admission,
which silently re-derived a plan without the recorded successor deployment;
the fence is now the only stage bit and a missing or foreign-bound plan fails
closed. Resumed transactions run the same lock admission as first admission,
except lock parents that are plan sources are never recreated so a deleted
source still fails loudly at copy. A committed snapshot that fails
post-rename validation is this transaction's disposable copy and restages
once. The bootstrap lock is taken before the marker re-read so repair or
adoption cannot republish identity in between, and the compatibility probe
no longer inherits NODE_OPTIONS.

Generated-by: Devin
Each boundary kept its own instanceof list, so new admission errors such as
root_migration_busy collapsed into internal_* codes. storageRootErrorDetail
is now the single mapping used by service management, setup, and update
reconciliation; a committed upgrade that fails only in post-commit cleanup
resolves the current root instead of reporting a recovery failure, and
readiness polling stops retrying roots that can never become ready.
Regression coverage pins root_requires_migration on retire, restart,
uninstall, configure, replace, and framed output over a legacy root.

Generated-by: Devin
The hand-enumerated path filter kept dropping files that participate in
root admission; switch to directory-level globs for the cli runtime-host
surface and the runtime-host client/server/operator trees.

Generated-by: Devin
…ence

Under the upgrade fence everything inside .maka-host is transaction-owned,
so a state/ directory without this transaction's completion record is debris
rather than foreign data: remove it and restage instead of wedging the root
forever on a torn cleanup. The committed-state check now also requires the
data and deployment directories to exist, since stageSnapshot always creates
both; a snapshot reduced to its completion record no longer commits empty
state. syncTree tolerates symlinks and other non-regular entries whose
dirent durability the parent directory sync already covers, and a marker
deleted mid-upgrade surfaces as root_unmarked rather than a raw ENOENT.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared mapping now also covers the activation and stdio-connect
boundaries, the four update command catches, update-check, and peer
management; activation and connect additionally preserve the authority
message that generalizedErrorMessage would have replaced with a fixed
fallback. Expected-target verification passes authority errors through
instead of folding a damaged or missing root into target_mismatch, and
service readiness fails fast for every permanent root code (unmarked,
not-found, invalid root, identity changed) instead of polling them out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The restage path now builds the replacement staging before removing state/,
so a failed restage leaves the verified snapshot as evidence instead of
turning a recoverable wedge into unrecoverable deletion. The completion
record attests whether the staged deployment carried an authority record,
and the committed check enforces it; a stray non-directory at state or the
staging path is debris rather than a wedge.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Host being started publishes the root marker itself, so service install
and start on a fresh or remounting root legitimately begin unmarked; failing
the readiness poll on root_unmarked/root_not_found raced the spawned Host's
first checkpoint and could roll back a healthy install. The connect boundary
now shares the one storage-root error recognizer instead of a private
instanceof.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XXL Over 2500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant