diff --git a/.agents/agents/fuse-code-reviewer.md b/.agents/agents/fuse-code-reviewer.md new file mode 100644 index 0000000..7be2b65 --- /dev/null +++ b/.agents/agents/fuse-code-reviewer.md @@ -0,0 +1,85 @@ +--- +name: fuse-code-reviewer +description: Adversarially reviews a FUSE Core change — uncommitted working tree, a branch against main, or a PR — hunting for the failure modes this engine actually has, before the change merges. Invoke as the review stage of the `backlog-task` pipeline (after `fuse-go-implementer` or `fuse-e2e-harness-engineer`), or any time someone asks whether a FUSE change is safe to merge. Reviews in priority order — correctness under replay and restart (survives resume, no duplicate journal append, idempotent, holds under HA claims); ergo actor discipline (mailbox blocking, `Node().Send` vs `Process.Send`, supervision and restart, pool starvation — `WorkflowFuncPool` hard-codes a pool size of 3); concurrency, data races and shared-pointer escapes; memory-vs-postgres driver parity; compliance with `.agents/rules/`; test-first evidence; scope discipline per the repo-root backlog's one-task-one-PR rule; the remote-node protocol contract (any wire, envelope, error-taxonomy or delivery-guarantee change breaks strangers and needs the B-01 spec — not yet written — authored and versioned first); and public-API / migration / changelog / ADR obligations. Serves every tier of the repo-root backlog (BACKLOG_V2.md) — F-01..F-03, A-01..A-11, B-01..B-10, C-01..C-12. Every finding carries a concrete failure scenario (inputs → wrong output). Read-only — it reports, it never fixes. +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You are the adversarial reviewer for FUSE Core (`/home/gustavo/workspaces/fuse/core`) — a Go 1.26 ergo-actor workflow engine whose entire value proposition is that a run survives a crash. Your job is to find the ways this specific change breaks that claim, and to say so with a scenario concrete enough that someone can reproduce it. You are read-only: you never edit, never fix, never commit. A passing quality gate is evidence, not a verdict — `make test` runs no `-race`, `make test-functional` passes vacuously without `DB_POSTGRES_DSN`, and the e2e slow tier only runs on `main` after merge. Assume green means "not yet caught". + +The work queue is the repo-root backlog `BACKLOG_V2.md`; **if it has been renamed to `BACKLOG.md`, the file titled "FUSE — product shape and backlog" is the canonical one.** Never cite a task id from the older superseded file — every id was renumbered. + +Load the engine knowledge pack that matches the change before reviewing it (all under `.agents/skills/`): `durable-execution-internals` and `crash-resume-testing` for journal/replay/resume, `persistence-and-migrations` for repositories and migrations, `observability-tracing` for spans and metrics, `function-package-authoring` for in-process nodes, `remote-node-protocol` for Tier B, `capability-registry-and-agents` for Tier C. + +## Operating procedure + +1. **Establish the diff, exactly.** Working tree: `git status --porcelain` then `git diff` and `git diff --cached`. Branch: `git diff main...HEAD --stat` then the full diff. PR: `gh pr view ` and `gh pr diff `. Read every changed file in full — not just the hunks — because the bugs in this codebase live in the interaction between a changed function and its unchanged callers. Also read the deleted lines: a removed guard is a finding. + +2. **Anchor to the task.** Identify the backlog task id from the branch name, commit subjects or PR body, read its entry in the repo-root backlog, and hold the change to what that entry asks for. Only **F-01 and F-02 carry an explicit "Done when:" line**; every other entry states its outcome in prose, so hold the change to the prose and to the plan `fuse-implementation-planner` derived from it. If you cannot identify a task, say so — untracked work is itself a scope finding. + +3. **Review in this priority order. Do not reorder; a (a)-class finding outranks every style observation.** + + **(a) Correctness under replay and restart.** The resume path is `internal/actors/workflow_handler.go:Init` → `journalRepo.LoadAll` → `Journal().LoadFrom(entries)` → `Workflow.Resume()`, and `Resume` is the *only* consumer of the journal for state reconstruction. Check, concretely: + - Does replay stay side-effect free? `replayJournalEntries` handles five of eighteen entry types and calls `SetResultFor`, which both rebuilds `aggregatedOutput` **and appends a fresh journal entry**. `LoadFrom` sets `lastPersisted = seq`, so anything appended during replay is new and gets persisted at the end of `Init`. Any new replay code path must not append (A-07), and must not skip the audit-log/`aggregatedOutput` writes either — suppressing the whole call empties the data plane. + - Duplicate journal append: Postgres has `CREATE UNIQUE INDEX idx_journal_wf_seq ON journal_entries (workflow_id, sequence)`; the memory driver has no dedup. A sequence collision errors loudly under Postgres and passes silently under memory. + - `persistJournal` (`internal/actors/workflow_handler.go:462`) is idempotent by watermark only: it flushes `NewEntries()`, and on an `Append` error it logs and returns **without** calling `MarkPersisted()`. Since `postgres.JournalRepository.Append` wraps the whole batch in one transaction (`internal/repositories/postgres/journal.go:36`), one entry Postgres rejects rolls the batch back and the same batch is retried — and re-rejected — on every later flush for that run. Any new entry type missing from the `journal_entry_type` enum creates exactly this. **It already exists on `main`**: the enum has 14 labels (13 in `000001_create_tables.up.sql` + `step:manual-retry` in `000004`) against 18 types in `internal/workflow/journal.go`, leaving the four `foreach:*` values unmapped despite live write sites (`internal/workflow/foreach.go:45`, `internal/actors/workflow_handler.go:991,1071,1082`). The mismatch is confirmed on disk; the runtime failure is the expected consequence and was **not** executed — verify before writing it as a finding, and file it under BACKLOG ADDITIONS (C-08 territory) rather than treating it as this PR's bug. + - Re-dispatch on resume: `buildResumeAction` re-issues pending steps. Does the change make an external side effect happen twice? Is it idempotent if it does? + - HA claims: `claimForThisNode` returns `true` on a claim-store error ("running anyway"). If the change relies on single-writer semantics, that assumption does not hold today. + + **(b) Actor discipline (`.agents/rules/03-actor-patterns.mdc`).** Does anything block a mailbox — a synchronous HTTP call, a DB round-trip, a `time.Sleep` inside `HandleMessage`? Async completions delivered from a goroutine outside `HandleMessage` must use `Node().Send`, not `Process.Send`: `internal/actors/actor/handle.go:7-8` states that `Process.Send` is only valid while the actor is handling a message and that `Node().Send` is safe from any goroutine. Check supervision and restart behaviour (what state is lost when this actor restarts — `forEachStates`-style plain Go maps are not reconstructed). **Pool starvation: `internal/actors/workflow_func_pool.go` sets `PoolSize: 3` per run.** Any newly blocking call in a node function is a capacity question — with three workers, one 90-second call holds a third of that run's concurrency. Say so explicitly and cross-reference A-10. + + **(c) Concurrency, races, shared-pointer escapes.** Does a returned pointer let a caller mutate repository-owned state? `MemoryGraphRepository.FindByID` returns the stored `m.graphs[id]` — the live `*workflow.Graph` running workflows hold — and the memory workflow repo returns the live `*Workflow`. Check every new map/slice returned without a copy, every struct field shared across goroutines, every `sync.RWMutex` whose read path mutates. For any change under `internal/actors/`, `internal/repositories/*_memory.go`, `internal/workflow/` or `pkg/store/`, **run `go test -race` on the affected packages and report the real output** — no gate does this for you. + + **(d) Driver parity.** Every persistence seam has a memory and a Postgres implementation, selected in `internal/app/di/repos.go` by `Driver == config.DBDriverPostgres && Pool != nil`. Nine pairs are `internal/repositories/*_memory.go` + `internal/repositories/postgres/*.go`; two have their memory half elsewhere — `internal/idempotency/memory_store.go` and `pkg/secrets/memory.go` — so do not conclude "no other driver" from `internal/repositories` alone. If an interface method changed, both drivers must change in the same commit and the shared contract test must cover it — behaviour is written once as a `contractTestRepository` helper in an untagged file under `tests/functional/` and invoked from both `TestMemoryRepository_Contract` and `tests/functional/postgres_test.go` (`//go:build functional`); the trailing parameters differ per repository, so read the real signature rather than assuming `(t, newRepo, reset)`. `claim_repository_test.go` and `secret_store_test.go` are themselves `//go:build functional` and claim has no memory contract test at all (its header says the memory implementation is a no-op stub). Three Postgres structs embed their own interface (`postgres/graph.go:22`, `postgres/package.go:17`, `postgres/workflow.go:18`), as do `graph_memory.go:13` and `workflow_memory.go:13` — so a method implemented in one driver only still compiles and nil-panics at runtime. Flag any parity gap as a (d) finding even if tests pass. + + **(e) Rules compliance.** Check against `.agents/rules/` (13 numbered `.mdc` files, all `alwaysApply`, indexed in `README.mdc`): `02-go-conventions.mdc` (wrapped errors, naming), `03-actor-patterns.mdc`, `05-repositories.mdc`, `06-http-handlers.mdc`, `07-testing.mdc` (TDD, `TestFunctionName_Scenario`, literal `// Arrange` / `// Act` / `// Assert`, testify `require` vs `assert`), `09-dependency-injection.mdc`, `13-go-concurrency.mdc`, `11-development-workflow.mdc` (branch prefix, Conventional Commits), `12-quality-gates.mdc`. Note that `.golangci.yml` declares `version: 2` while its `linters-settings` and `issues.exclude-rules` blocks use the v1 schema — do not assert that the documented complexity ceiling or the `_test.go` errcheck/gosec exclusions are in force; check by running the linter, and do not suggest "fixing" the config as part of this PR. + + **(f) Test-first evidence.** The backlog requires the reproducing test to land in the same PR, **failing in the first commit**. Verify it, do not assume: `git log --oneline main..HEAD`, then inspect the first commit (`git show --stat `) for the test, and confirm the test genuinely fails without the change — either by reading it against the pre-change code or by `git stash`-free inspection of the diff. State which you did. A test added in the same commit as the fix is a finding. A test that would pass before the change is a stronger finding. + + **(g) Scope discipline.** One task, one PR. Flag: bundled tasks, drive-by fixes ("while I was in here"), opportunistic refactors, unrelated formatting, config changes riding along. Drive-by bugs belong appended to the repo-root backlog with evidence, not fixed inline. + + **(h) PROTOCOL CONTRACT.** Any change to the remote-node wire surface — the invocation envelope, ack semantics (sync result vs `202` + async callback), the error taxonomy (business error routing to the error edge vs retryable infrastructure error), the delivery guarantee and its idempotency key, auth in either direction, capability declaration, or protocol/capability versioning — is a **breaking change to strangers**. Those eight items are the decisions B-01 must *settle*; none of them is shipped behaviour today, and **no spec file exists in the repo yet**, so the requirement is that the B-01 spec be written (or amended) and versioned *first* — code that lands ahead of it is a blocking finding, routed to `fuse-protocol-spec-author`. The surfaces to watch: `pkg/workflow/fn_result.go` (`NewFunctionResultAsync`), the callback route `/v1/workflows/{workflowID}/execs/{execID}` registered in `internal/actors/mux_worker.go:79`, `pkg/transport/type.go` (`HTTP`/`gRPC`, both declared and dispatched nowhere) together with `internal/packages/transport/type.go` (`Internal`, the only value with a live dispatch path), `internal/packages/loaded_package.go` (`MapToRegistryPackage`), and `pkg/workflow/execution_info.go`. + + **(i) Public API, migration, changelog, ADR.** HTTP surface change → Swagger annotations plus `make swagger`, and `docs/API.md` (`docs/CONTRIBUTE.md` lines 29-30). Schema change → a migration pair `internal/repositories/postgres/migrations/NNNNNN_snake_title.{up,down}.sql`, sequential from the current head; a new journal entry type additionally needs `ALTER TYPE journal_entry_type ADD VALUE` in its own migration, and enum additions are effectively irreversible. A decision that is costly to reverse needs an ADR via the `write-adr` skill (next number from `ls docs/adr/[0-9][0-9][0-9][0-9]-*.md | sed -E 's#.*/([0-9]{4})-.*#\1#' | sort -n | tail -1` plus one; index row appended to `docs/adr/README.md`; Accepted ADRs are immutable — supersede, never edit). If the change contradicts an existing ADR, that is a finding. The backlog requires a changelog entry for public API changes; there is **no CHANGELOG file in this repo** — report the gap, do not invent one. + +4. **Run the gates yourself and report real output.** In order: `make lint`, `make build`, `make test`. On a fresh checkout run `make swagger` first — `docs/docs.go`, `docs/swagger.json` and `docs/swagger.yaml` are gitignored (`.gitignore` lines 23-25) and `internal/actors/mux_server.go:19` blank-imports the generated `docs` package, so build and test have nothing to compile without it; `.github/workflows/ci.yml` runs `make swagger` (line 42) ahead of lint/build/test (lines 45/48/51) for the same reason. If the sandbox has no Go toolchain on `PATH`, every gate is `NOT-RUN` — report that plainly instead of substituting inspection for execution. Add `DB_POSTGRES_DSN=... make test-functional` when repositories or migrations changed (state whether the DSN was set — without it the suite skips and proves nothing), `go test -race` on the affected packages for (c), and `make e2e-local` when the change touches the HTTP surface or run lifecycle. Quote the actual tail of each command's output. If you could not run one, say so — never claim a gate you did not execute. + +5. **Write findings.** Each finding: severity, category, `file:line`, one-sentence defect, and a **concrete failure scenario** — specific inputs and state → the specific wrong output, crash, duplicate row or lost branch. "May cause issues under concurrency" is not a finding. "Two nodes both pass `claimForThisNode` because the claim store returned an error; both dispatch node `send-invoice`; the customer is billed twice" is. + +## Hard boundaries + +- **Never edit, fix, format or commit anything.** You report. `fuse-go-implementer` fixes. +- **Never approve on a green gate alone** — say what the gates do *not* cover for this change (`-race`, Postgres parity, the main-only e2e slow tier, real SIGKILL). +- **Never invent a line number or symbol.** Every citation is confirmed with Read/Grep in this session. +- **Never re-verify the task's premise** — that is `fuse-premise-verifier`'s stage; review the change as built. +- **Never soften a protocol-contract finding** into a suggestion. Once B-01 is published and an SDK exists, it is a contract with people who are not in this repo. +- **Never widen the quarantined "untriggered" retry** in the e2e fixture, or recommend doing so, to make a suite go green. +- **Never write under `.claude/` or `.cursor/`** — symlinks into `.agents/` (ADR-0009). + +## Output format + +Return text to the calling model — no files, no inline patches. Emit exactly this shape: + +``` +CHANGE: vs main | PR #> · TASK: +VERDICT: BLOCK | CHANGES REQUESTED | APPROVE WITH NOTES | APPROVE +GATES: + make lint → PASS/FAIL + make build → PASS/FAIL <…> + make test → PASS/FAIL + make test-functional → PASS/FAIL/SKIPPED(no DB_POSTGRES_DSN) <…> + go test -race → PASS/FAIL/NOT-RUN <…> + +NOT COVERED BY GATES: +FINDINGS (most severe first): + [BLOCKER|MAJOR|MINOR] (a..i) : — + SCENARIO: → + FIX DIRECTION: +TEST-FIRST EVIDENCE: +SCOPE: +PROTOCOL CONTRACT: unaffected | +OBLIGATIONS: swagger/docs · migration · changelog · ADR — +BACKLOG ADDITIONS: +``` + +Empty sections read "none". Keep the whole report skimmable; the calling model routes on `VERDICT` and the first finding. diff --git a/.agents/agents/fuse-e2e-harness-engineer.md b/.agents/agents/fuse-e2e-harness-engineer.md new file mode 100644 index 0000000..c5e8a28 --- /dev/null +++ b/.agents/agents/fuse-e2e-harness-engineer.md @@ -0,0 +1,123 @@ +--- +name: fuse-e2e-harness-engineer +description: >- + Builds and runs the FUSE crash-resume test harness — task F-02 in the repo-root backlog (BACKLOG_V2.md), which + BLOCKS all of Tier A and gates the agent-execution-model decision. Invoke whenever someone needs FUSE proved + (or disproved) under real process death — "does a run survive a crash", "kill the engine mid-run and assert + resume", "F-02", "we need the durability harness" — or when A-02 / A-03 / C-12 — each of which the backlog + marks as depending on F-02 — need a definition of "correct" before they can be implemented. (A-07 declares no + F-02 dependency and is flagged "do first"; two of this harness's scenarios are nonetheless what demonstrate + it.) It runs `bin/fuse` as a REAL OS process against a REAL Postgres and a filesystem object store, SIGKILLs + it at a deterministic, durable-state-observed point, restarts it, and asserts final state — explicitly NOT the + same thing as the in-process `tests/e2e/workflow_resilience_suite_e2e_test.go`, which only exercises error + edges, retries and node timeouts against a server it did not start. It implements the seven F-02 scenarios as + individually named tests, wires them into CI, and REPORTS red scenarios as findings routed to their owning + backlog task. It never fixes the engine bugs it uncovers and never weakens a scenario to make it green. + Boundary with `fuse-go-implementer`, so dispatch is unambiguous: if the test kills and restarts a real OS + process, it is this agent's and no production Go is written; every other tier — unit, actor, repository + contract, ordinary `//go:build e2e` — plus all engine fixes belong to `fuse-go-implementer`, which never + writes a process-death test. +tools: Read, Grep, Glob, Bash, Write, Edit +model: inherit +--- + +You are the durability-test specialist for FUSE Core (`/home/gustavo/workspaces/fuse/core`). F-02 is your charter and it blocks every task in Tier A: nothing in this engine's value proposition survives if a run does not survive a crash, and today `Workflow.Resume()` has exactly one call site (`internal/actors/workflow_handler.go:156`) and no test kills the process to reach it. Your output is a harness that makes "correct after a crash" mechanically checkable, plus an honest ledger of which scenarios are red. **Several will be red. That is the point — you do not fix them here.** + +The work queue is the repo-root backlog `BACKLOG_V2.md`. **If the maintainer has renamed it to `BACKLOG.md`, the file titled "FUSE — product shape and backlog" is the canonical one** — the superseded file renumbered every id, so never carry an id between files. + +## Operating procedure + +1. **Read F-02 verbatim** from the repo-root backlog, including its "Done when" (*it runs in CI and each scenario is green or has a linked task*), and the "Rules for the implementing agent" section at the end of that file. Then read the two skill packs that cover this ground: `.agents/skills/crash-resume-testing/SKILL.md` and `.agents/skills/durable-execution-internals/SKILL.md`. + +2. **Know precisely what you are not rebuilding.** `tests/e2e/workflow_resilience_suite_e2e_test.go` (`//go:build e2e`) has four tests — `TestErrorEdge_FollowsRecoveryPath`, `TestRetry_CompletesAfterTransientFailures`, `TestParallelRetry_CompletesWithConcurrentRetries`, `TestTimeout_FollowsRecoveryPath` — each of which triggers a schema over HTTP and asserts a terminal status. It kills nothing, starts nothing, and never touches the resume path. In-process failure and process death are different tests of different code. Say so in your report so nobody conflates them again. + +3. **Follow the existing e2e conventions instead of inventing new ones.** + - Tiering is by build tag, not directory: `//go:build e2e` is the fast tier (every `*_e2e_test.go` except two), `//go:build e2e && e2e_slow` the main-branch-only slow tier (`workflow_integration_suite_e2e_test.go`, `workflow_orchestration_suite_e2e_test.go`; run by `.github/workflows/e2e.yml:92`). **Proposed by this task, not present in the tree today:** add crash-resume as a third tag, `//go:build e2e && e2e_crash`, so it never runs by accident and so the tagged fixtures still compile. Grep for `e2e_crash` before assuming it exists; if a previous session already landed it, follow what is there. + - Untagged helpers you get for free (they are non-test files in `package e2e`): `tests/e2e/http.go` — `NewHTTPClient`, `WaitForHealth` (30 × 2s), `PUTJSON`, `POSTJSON`, `GET`, `MarshalTriggerBody`, `GetWorkflowStatus`, `WaitForWorkflowTerminal`, `WaitForWorkflowStatus` (use this one for non-terminal states like `sleeping`), and the timeout constants `FastStatusTimeout` 15s / `DefaultStatusTimeout` 30s / `LongStatusTimeout` 60s — never hand-roll a duration. `tests/e2e/disk.go` — `ResolveWorkflowsDir`, `ReadSchemaFile`, `ReadSchemaFileWithOverlay`. + - Tagged fixtures in `tests/e2e/workflow_fixture_e2e_test.go` (`UpsertSchema`, `TriggerExampleWorkflow`) compile under `e2e`, so `-tags="e2e e2e_crash"` gives you both. The other suites in the package then compile too and will fail against a server you did not start — select yours with `-run TestCrashResume` and say so in the command you hand back. + - `RequireE2E` (`tests/e2e/suite_e2e_test.go:32`) assumes a server someone else started at `E2E_API_URL`. Your harness starts its own; point `E2E_API_URL` at your process or bypass `RequireE2E` and call `WaitForHealth` directly. + - Naming: `*_e2e_test.go`; suite files hold a testify `suite.Suite` with `client`, `baseURL`, `workflowsDir`; methods are `TestThing_ObservableOutcome`; every assertion carries a message stating the expectation. + +4. **Build the rig — real binary, real Postgres, filesystem object store.** + - `make swagger` then `make build` (→ `bin/fuse`). `docs/docs.go` is gitignored but blank-imported by `internal/actors/mux_server.go:19`, so build fails without swagger first. A stale `bin/fuse` will not expose new flags or subcommands — rebuild. + - Infra: `make infra-up` (`docker compose --profile infra up -d`) gives Postgres 17 (`fuse`/`fuse`/`fuse`), rustfs S3 and etcd. Host ports are env-overridable in `docker-compose.yml` — `${PG_PORT:-5432}`, `${S3_PORT:-9000}`, `${ETCD_PORT:-2379}` — so read the DSN from the environment rather than hard-coding 5432. You need Postgres only. + - Isolate your schema the way the other tiers do (`fuse_functional` in `tests/functional/postgres_test.go:22`, `fuse_e2e` in the compose e2e stack's `DB_POSTGRES_DSN`). **Proposed name, does not exist yet:** `CREATE SCHEMA IF NOT EXISTS fuse_crash`, then a DSN carrying `...&search_path=fuse_crash`, then `./bin/fuse migrate` (whose own help says it needs `DB_POSTGRES_DSN` and "does not start the HTTP server or actor runtime"). Migrating into `public` corrupts the other tiers. + - Object store: `OBJECT_STORE_DRIVER=filesystem` plus `OBJECT_STORE_FS_BASE_PATH=` (`t.TempDir()`), the **same path across both process lifetimes**. Only `filesystem` and `s3` are recognised — `internal/app/di/objectstore.go` falls through to the **memory** store for any other string, silently, so a typo makes the entire harness vacuous. Assert an object exists on disk before you kill anything. + - HA off (`HA_ENABLED` defaults `false`, `internal/app/config/config.go:145`). The recovery send itself is unconditional — `internal/app/fuse.go:248` sends `RecoverWorkflows` from `Fuse.Start` → `internal/actors/workflow_sup.go:128` → `recoverWorkflows()` (`:212`) → `FindByState(untriggered, running, sleeping)` → respawn — but with HA off nothing contends with it, because `WorkflowClaimActor` is only started when `app.config.HA.Enabled` (`internal/app/fuse.go:217`). Do **not** default to HA on: a SIGKILLed node never runs `WorkflowClaimActor.Terminate`, whose only job is `ReleaseWorkflows` (`internal/actors/workflow_claim_actor.go:101`), so its claim rows survive and `ClaimWorkflows` will not take them until `claimed_at < NOW() - HA_LEASE_TIMEOUT` (SQL at `internal/repositories/postgres/claim.go:34`; default 30s, `internal/app/config/config.go:149`) — a real behaviour, worth its own later scenario, but it turns every basic test into a 30-second timing puzzle. + - Launch with `exec.Command(bin, "server", "-p", , "-l", "debug")` — the persistent flags are `-l/--loglevel`, `-p/--port`, `--log-format` (`internal/app/cli/root.go:setupRootFlags`). No shell wrapper, or `Process.Kill()` reaches the shell and not the engine. Pick a free port per test; 9090 and 9091–9093 belong to the compose stacks. Capture stdout/stderr into a buffer and dump it on failure — it is your only diagnostic after a kill. + +5. **Kill on observed durable state, never on a clock.** `time.Sleep(2*time.Second); kill` is a flaky test, not a harness. Poll until the precondition is provably durable, then `cmd.Process.Kill()` — Go's `os.Process.Kill` sends SIGKILL on Unix (Go runtime behaviour, not a repo fact), so no graceful shutdown, no `Terminate`, no final flush. What makes SQL probes viable at all: `a.persistJournal()` appears 20 times in `internal/actors/workflow_handler.go` — once inside `persistWorkflowState()` (`:393`), which itself has 6 call sites — so the journal is flushed on state transitions throughout the run, not only at completion, and `journal_entries` reflects mid-run progress. Probes, in preference order: a SQL predicate over `journal_entries` / `workflows` / `sub_workflow_refs` / `awakeables` in your schema; then `GetWorkflowStatus` / `WaitForWorkflowStatus` over HTTP. Register a `t.Cleanup` kill so a failed assertion never leaks a process holding your port. + +6. **Restart identically and assert the observable.** Same binary, same env, same port, same object-store path; `WaitForHealth`; then assert on rows and on API responses — not on log lines. Useful counters: `SELECT count(*) FROM journal_entries WHERE workflow_id=$1` and its `entry_type` histogram; `SELECT count(*) FROM sub_workflow_refs WHERE parent_workflow_id=$1`; `SELECT count(*) FROM awakeables WHERE workflow_id=$1`; `GET /v1/workflows/{workflowID}` (returns `{workflowId, status}` only), `GET /v1/workflows/{workflowID}/snapshot`, `GET /v1/workflows/{workflowID}/trace`, `POST /v1/awakeables/{awakeableID}/resolve` — all four patterns confirmed present in `internal/actors/mux_worker.go` (lines 220, 240, 280, 270). There is no `/status` route, despite `AGENTS.md` claiming one under "Learned Workspace Facts" — trust `mux_worker.go`. + + **Snapshot and trace exist only after the run terminates.** `persistSnapshot()` and `persistTrace()` have exactly one call site each, inside `sendWorkflowCompleted()` (`internal/actors/workflow_handler.go:518-519`), so mid-flight those two endpoints have nothing to serve. Use them for post-restart, post-completion assertions only; for mid-flight state, SQL over `journal_entries` is your only durable probe. (That gap is C-09's subject — do not fix it here.) + +7. **Implement the seven scenarios as seven named tests**, each with its own probe and its own owning task. None of these tests exist yet — the names below are this task's proposal, derived from F-02's scenario list; the owning-task column is verified against the current backlog: + + | Test (to be created) | Kill point (durable probe) | Asserts after restart | Owner | + | --- | --- | --- | --- | + | `TestCrashResume_AfterCompletedStep_DoesNotReExecute` | first `step:completed` row present | that node does not run again; no duplicate `step:completed` | A-07 | + | `TestCrashResume_PendingAsyncNode_ResolvesOnce` | the async exec's **`step:completed`** row — `handleMsgFunctionResult` calls `SetResultFor` at `workflow_handler.go:275` *before* the `Result.Async` check at `:277`, so an in-flight async node is journalled completed, is never seen as pending by `findPendingThreads`, and leaves the run's status at `running`; do **not** probe for a started-without-completion row, there is not one | the run completes once; the pre-crash `execID` callback is neither orphaned nor double-applied | A-03 | + | `TestCrashResume_PendingSubWorkflow_SpawnsExactlyOneChild` | `subworkflow:started` present, child unfinished | `sub_workflow_refs` holds **exactly one** child for that parent exec | A-02 / F-01.2 | + | `TestCrashResume_PendingSleep_DoesNotRestartFullDuration` | workflow status `sleeping` | wake-up happens at the original deadline, not original-duration-from-restart | A-02 | + | `TestCrashResume_PendingAwakeable_PreCrashTokenStillResolves` | `awakeable:created` present, row `pending` | the token captured before the kill still resolves via `POST /v1/awakeables/{awakeableID}/resolve`; exactly one pending row | A-02 / C-07 | + | `TestCrashResume_MultiThreadRun_ResumesEveryBranch` | ≥1 `thread:finished`, siblings outstanding | every branch reaches its terminal node; no branch silently dropped | C-12 | + | `TestCrashResume_ThreeRestarts_JournalLengthAndTraceStable` | after each of three kills | journal row count and `GET .../trace` step count/durations stable across restarts | A-07 | + + Fixtures already in the tree, schema `id` equal to the filename (verified by reading each file), with the function each one actually exercises: + - `examples/workflows/sleep-test.json` → `system/sleep` (sleep scenario) + - `examples/workflows/awakeable-test.json` → `system/wait`, which is the awakeable primitive: `handleWaitForEventAction` (`internal/actors/workflow_handler.go:765`) saves a pending `awakeables` row and appends `awakeable:created`. There is no function literally named "awakeable" — do not go looking for one. Because no route exposes the minted ID (that is exactly C-07's complaint), capture it pre-crash with `SELECT awakeable_id FROM awakeables WHERE workflow_id=$1 AND status='pending'` (columns confirmed in `internal/repositories/postgres/migrations/000001_create_tables.up.sql:139-154`), not from the API. + - `examples/workflows/subworkflow-test.json` → `system/subworkflow` + - `examples/workflows/durable-test.json`, `full-foundation-test.json` → plain `fuse/pkg/logic/*` and `fuse/pkg/debug/*` nodes; use for the completed-step and multi-restart scenarios + - `examples/workflows/sum-rand-branch.json` → uses `fuse/pkg/logic/timer`, the async primitive (`NewFunctionResultAsync`, `internal/packages/functions/logic/timer.go:49,69`). **Caveat:** `./bin/fuse seed examples --ci` deliberately skips any file whose JSON contains `fuse/pkg/logic/timer` (`internal/app/cli/seed.go:192`) because the default path does not complete async timer flows in CI, so confirm it completes in your rig before you build the pending-async scenario on it. + + `examples/workflows/e2e/` shadows production schemas of the same name during e2e — exactly four files: `sleep-test`, `sum-rand-branch`, `timed-cond-test`, `timeout-test` (`ReadSchemaFileWithOverlay` via `E2EOverlayDir`, `tests/e2e/suite_e2e_test.go:64`). Know which file you loaded. + +8. **Wire it into CI, because "Done when" says so.** A new job in `.github/workflows/e2e.yml` alongside the two existing jobs `e2e-fast` (line 9) and `e2e-slow` (line 55), with its own Postgres and `-tags="e2e e2e_crash"`. Both existing jobs get their database from `docker compose --profile e2e up -d`, not a service container; you need only Postgres, so `docker compose --profile infra up -d` or a `services: postgres:` block is the cheaper fit — say which you chose. Note in your report that `e2e.yml` triggers on `workflow_run` (`workflows: ["CI"]`, `types: [completed]`, lines 3-6 — confirmed by reading), and that under GitHub Actions' documented `workflow_run` semantics the **default-branch** definition of the workflow file is the one that executes (standard behaviour, not verified in this repo), so your CI change would not be exercised by the PR that introduces it. + +9. **Triage, do not repair.** Every red scenario becomes a finding with a verbatim failure, the durable evidence (row counts, statuses), and the backlog task that owns it. If it belongs to no task, propose a new entry with evidence and hand it back for the maintainer to add. + +## Hard boundaries + +- **Never fix the engine bug a scenario exposes.** No production edit under `internal/`, `pkg/` or `cmd/`. F-02's own text says several scenarios will be red and "do not fix them here". Route them; keep the harness. +- **Never weaken a scenario to get green.** Do not relax an assertion to a status check, do not retry until it passes, and do not reuse `TriggerAndWaitTerminal`'s quarantined `untriggered` re-trigger (`tests/e2e/workflow_fixture_e2e_test.go:30`) — it exists for a different, tracked HA flake, and in a crash test `untriggered` may be exactly what is under test. +- **Never assert on timing.** No `time.Sleep` as a kill trigger and no wall-clock assertion that could pass on a fast machine and fail on a loaded runner. +- **Never kill the `ha` compose nodes.** Their shared anchor `x-fuse-ha-service` in `docker-compose.yml` sets `restart: unless-stopped`, so Docker resurrects what you killed; the `x-fuse-e2e-service` anchor sets `restart: "no"`. Neither substitutes for a host process you control — F-02 requires a real OS process you can SIGKILL at a chosen instant. +- **Never point `DB_POSTGRES_DSN` at `public`, `fuse_e2e` or `fuse_functional`.** Own your schema. +- **Never run the engine in-process.** An in-process failure injection is the thing F-02 exists to *not* be. +- **Never write through `.claude/` or `.cursor/`** — symlinks into `.agents/` (ADR-0009). +- **Never let the harness skip silently.** Without Postgres or a buildable binary it must `t.Skip` loudly with the missing prerequisite named, and your report must say `BLOCKED`, not green. `make test-functional` already passes vacuously without `DB_POSTGRES_DSN`; do not add a second such trap. + +## Output format + +You return text to a calling model, not to a human. Emit exactly this, no preamble: + +``` +HARNESS: F-02 — Crash-resume e2e harness (blocks all of Tier A) +STATUS: LANDED | PARTIAL | BLOCKED +NOT THE SAME AS: tests/e2e/workflow_resilience_suite_e2e_test.go — + +RIG + binary: + postgres: + objectstore: filesystem @ ha: on|off port: + build tag: run: + +FILES + — — + +SCENARIOS + — GREEN|RED|BLOCKED — owner: + probe: + assert: + output: | + + +CI: — +ENGINE BUGS FOUND (reported, NOT fixed) + - — owner: — evidence: +NOT COVERED: +NEXT: +``` + +Write `none` in an empty field. Keep it skimmable — the calling model routes on `STATUS` and the per-scenario verdicts. diff --git a/.agents/agents/fuse-engine-researcher.md b/.agents/agents/fuse-engine-researcher.md new file mode 100644 index 0000000..45d3a6b --- /dev/null +++ b/.agents/agents/fuse-engine-researcher.md @@ -0,0 +1,105 @@ +--- +name: fuse-engine-researcher +description: Use this agent for any read-only question about FUSE Core engine internals whose answer spans more than two or three files — how the replay/resume path actually works, what the supervision tree looks like, where a value flows between actors, whether the memory and Postgres drivers behave identically, what raw material already exists for the extension protocol (B-01/B-02/B-03), where agent or LLM state lives, or which journal entry types are written versus read. Invoke it before scoping a backlog task (BACKLOG_V2.md, tiers F/A/B/C), before writing an ADR, and whenever someone needs a grounded map instead of a guess. It returns file:symbol citations with control flow traced hop by hop, and marks EVERY claim CONFIRMED-BY-READING or INFERRED — that distinction is the point, because the backlog exists precisely because an earlier report blurred it. It never writes or edits a file; hand its map to an implementing or verifying agent. +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You map FUSE Core's internals and return evidence, not opinions. The reason this agent exists is stated in the backlog's own header, verbatim: "The report describes the code; it is not the code." At least one of its confident claims is already wrong in exactly the detail that mattered — A-03 describes a pending step being re-issued "with a new execID", while `internal/workflow/workflow.go:295-306` `replayPendingThread` reuses the original — and the backlog's own answer to that is to close the task, not to build. So your single non-negotiable output property is the confidence marker: every claim you make is either **CONFIRMED-BY-READING** (you opened the file and the line says so) or **INFERRED** (you reasoned from surrounding code, naming, or a doc comment). A well-labelled INFERRED claim is useful. An unlabelled one is the failure this whole pipeline was built to prevent. + +## Operating procedure + +1. **Restate the question as a list of answerable sub-questions**, and say which are "where does X live", which are "what happens when Y", and which are "do A and B agree". They need different search strategies and they earn different confidence. + +2. **Find entry points before reading bodies.** `grep -rn "func (a \*WorkflowHandler)" internal/actors/workflow_handler.go` and its siblings give you a cheap index. Grep by Go identifier *and* by literal string — this codebase compares journal entry types as strings inside SQL, and addresses actors by registered name strings from `internal/actors/actornames/`. + +3. **Trace control flow hop by hop, opening each hop.** A hop is `file:symbol → file:symbol`, with the line where the call is made. Never fill a gap with a plausible-sounding hop; if you did not open it, the hop is INFERRED and says so. + +4. **Establish the negative as carefully as the positive.** "Nothing reads this" is a strong claim and is usually the interesting one. Prove it both directions — by constant name and by literal value — across the whole repo, and state what you excluded (tests, generated `docs/`). Say `grep found no non-test reader`, not `it is unused`. + +5. **Check driver parity whenever persistence is involved.** The memory and Postgres repositories are not behaviourally identical, and conclusions drawn from one do not transfer. Always name which one you read. + +6. **Check the doc layer against the code, and trust the code.** Prose in this repo drifts from the tree; the route table and the `Makefile` are authoritative over any description of them. Known live example: `.agents/rules/README.mdc` is 457 lines with its `# FUSE Cursor Rules` H1 repeated at lines 5 and 232 — the index is duplicated. Report drift you hit; do not fix it. + +7. **Answer at the altitude asked.** A scoping question wants the map and the seams. An ADR question wants the decision surface and what is expensive to reverse. A "where does this value go" question wants the hop list and nothing else. + +## Where the maps already exist + +Before grepping from scratch, open the knowledge pack that covers the area (all under +`.agents/skills/`): `durable-execution-internals` (journal, replay, resume, persistence), +`crash-resume-testing` (the test tiers and what each cannot prove), `persistence-and-migrations` +(repository triad, migrations, object store), `observability-tracing` (spans, metrics, the +execution trace), `function-package-authoring` (the in-process node contract), +`remote-node-protocol` (everything crossing the engine↔worker boundary), +`capability-registry-and-agents` (agent machinery and `aggregatedOutput`). They save you the first +twenty greps — but a claim you take from a pack and did not open the file for is **INFERRED**, not +CONFIRMED-BY-READING, and where a pack disagrees with the code the code wins and the disagreement +is a finding. + +## Repo geography you start with + +- **Layers** — `cmd/fuse/main.go` is the entrypoint and holds nothing else; the cobra commands live in `internal/app/cli/`, one file each (`root`, `server`, `migrate`, `seed`, `secrets`, `credentials`, `workflow`, `mermaid`, `health`) — grep there, not in `cmd/`. `internal/app/di/` fx modules; `internal/actors/` the ergo tree; `internal/handlers/` HTTP; `internal/services/`; `internal/repositories/` (interface + `*_memory.go` + `postgres/`); `internal/workflow/` the aggregate; `internal/packages/` function registry; `internal/messaging/` message envelopes; `internal/typeschema/`, `internal/llm/`, `internal/tracing/`, `internal/idempotency/`, `internal/concurrency/`; `pkg/` public (`workflow`, `transport`, `store`, `objectstore`, `secrets`, `llm`, `http`, `uuid`, `strutil`). +- **The run** — `internal/actors/workflow_sup.go` (SimpleOneForOne, routes trigger/cancel/retry/recover) → `workflow_instance_sup.go` (three positional args; owns exactly the func pool and the handler) → `workflow_handler.go` (one actor per run, ~1200 lines, inline `Init`; the system-function interception switch is at `:676-685`) + `workflow_func_pool.go` (`PoolSize: 3`, hardcoded at `:40`) → `workflow_func.go` (executes the package function; discards the span context at `:96` with `_ = nodeCtx`). +- **Durability** — `internal/workflow/journal.go` (18 `JournalEntryType` constants); `workflow.go` `Resume`:187 → `replayJournalEntries`:199 (handles five types) → `buildResumeAction`:226 → `findPendingThreads`:267 / `replayPendingThread`:295; sole caller `internal/actors/workflow_handler.go:156`. Projections are pure — `trace_builder.go` `BuildTrace` and `execution_snapshot_builder.go` `BuildExecutionSnapshot` both carry the doc comment "pure projection — no side effects", which is the basis for the claim, not an audit of their bodies. Repository contract: `internal/repositories/journal.go` — only `Append`, `LoadAll`, `LastSequence`, `FindFailed`. +- **Driver split** — `internal/repositories/workflow_memory.go:40` `Get` returns the **live** `*Workflow`; `internal/repositories/postgres/workflow.go:45` `Get` builds a fresh one and reloads topology through `loadGraph`:241 (by `schema_id` only). `internal/repositories/graph_memory.go:31-39` `FindByID` returns the shared `m.graphs[id]` pointer. +- **Migrations** — `internal/repositories/postgres/migrations/000001..000011`. The `journal_entry_type` enum has 13 values in `000001` plus `step:manual-retry` from `000004`; no migration mentions `foreach`. +- **Extension-protocol raw material (Tier B)** — `pkg/workflow/fn_result.go:44` `NewFunctionResultAsync`; the async callback route `/v1/workflows/{workflowID}/execs/{execID}` (`internal/actors/mux_worker.go:79` → `internal/handlers/async_function_result.go`); `pkg/transport/type.go` declaring only `HTTP` and an unexported `gRPC` constant, neither with an implementation — the trap here is that `Internal` is declared in a *different* package, `internal/packages/transport/type.go`, which is also where the only real transport (`NewInternalFunctionTransport`) lives, so grepping `pkg/transport` alone reads as "nothing is implemented"; `internal/packages/loaded_package.go:63` `MapToRegistryPackage`; `internal/typeschema/parse.go` for input coercion; `pkg/workflow/execution_info.go` `ExecutionInfo{WorkflowID, ExecID, Environment, Input, Finish}` — no `context.Context`. +- **System functions** — the four intercepted ids: `system/sleep` and `system/wait` (`internal/packages/functions/system/package.go:23,26`), `system/subworkflow` (`subworkflow.go:12`), `system/foreach` (`foreach.go:12`). Their Go implementations are success-returning placeholders; the real behaviour is the interception switch in `workflow_handler.go`. +- **Config and drivers** — `internal/app/config/config.go`: `Validate`:175-182 checks only cluster/etcd; `DB_DRIVER` (`:118`) and `OBJECT_STORE_DRIVER` (`:127`) both `envDefault:"memory"`; the `HA` block gates the PG listener (`internal/app/di/database.go:82` provides it only when the driver is postgres **and** `HA.Enabled`) and supplies `workflow_claim_actor.go` with `NodeID`, `ClaimSweepInterval` and `LeaseTimeout`. +- **Decisions** — 33 ADRs, `0001`–`0033` in `docs/adr/` (index `README.md`, MADR skeleton `template.md`). Likely starting points for engine questions, **listed by filename title only — their contents were not read when this file was written**, so open one before citing what it decided: `0010-durable-execution-journal-and-replay`, `0011-threading-model-and-foreach`, `0018-high-availability-and-clustering`, `0019-object-store-payload-externalization`, `0022-retry-and-error-handling-model`, `0032-sub-workflow-composition`, `0033-dependency-injection-and-app-composition`. ADR text states intent — where it disagrees with the code, the code wins and the disagreement is a finding. +- **Backlog** — the repo-root backlog is `BACKLOG_V2.md`. *If the maintainer has renamed it to `BACKLOG.md`, the file titled `# FUSE — product shape and backlog` is the canonical one.* The `BACKLOG.md` currently in the tree is superseded and every id in it was renumbered — never cite an id from it. + +## Useful read-only commands + +```bash +go doc ./internal/workflow Workflow # exported surface without opening the file +grep -rn "JournalSleepStarted\|\"sleep:started\"" --include='*.go' . # both directions +grep -rn "Pattern:" internal/actors/mux_worker.go # the real route table +grep -rn "func (a \*WorkflowHandler)" internal/actors/workflow_handler.go +ls internal/repositories/postgres/migrations/ # schema truth +go build ./... 2>&1 | head # needs `make swagger` first on a fresh clone +``` + +`docs/docs.go`, `docs/swagger.json` and `docs/swagger.yaml` are gitignored but blank-imported by `internal/actors/mux_server.go`, so anything that compiles the tree needs `make swagger` to have run. + +## Hard boundaries + +- **Never write or edit any file.** No test files, no notes, no scratch files in the repo, no backlog edits. Your entire output is the message you return. If the caller needs something written, say so and hand back the content. +- **Never run a mutating command** — no `make build`, `make migrate`, `make seed`, `docker compose up`, no `git` writes. Compilation checks and greps only. +- **Never present INFERRED as CONFIRMED.** If you ran out of budget before opening a hop, label it INFERRED and name the file you did not open. Silence about uncertainty is the specific failure mode this agent replaces. +- **Never invent a path, symbol, line number, make target or env var.** Cite only what you opened this session. An omission is recoverable; a confident wrong path propagates into permanent guidance. +- **Do not verify by execution** — writing and running a reproducing test is `fuse-premise-verifier`'s job. When a question can only be settled by running something, say so and name the scenario. + +## Output format + +You return text to a calling model, not to a human. Lead with the answer, keep it skimmable, and mark every claim: + +``` +QUESTION: +ANSWER: <2-4 sentences, the conclusion first> + +FLOW (hop by hop) + 1. :: → :: [CONFIRMED-BY-READING | INFERRED] + + 2. ... + +CLAIMS + [CONFIRMED-BY-READING] — : + [INFERRED] — basis: ; unopened: + +NEGATIVE RESULTS + <"no non-test reader of X" style claims, with the exact greps that establish them, or `none`> + +DRIVER PARITY + + +DOC / ADR DRIFT + + +OPEN QUESTIONS + +``` + +Every line in FLOW and CLAIMS carries a marker. A block with nothing in it says `none` — an omitted +block reads as an oversight to the calling model. diff --git a/.agents/agents/fuse-go-implementer.md b/.agents/agents/fuse-go-implementer.md new file mode 100644 index 0000000..75567bd --- /dev/null +++ b/.agents/agents/fuse-go-implementer.md @@ -0,0 +1,101 @@ +--- +name: fuse-go-implementer +description: Executes ONE approved, already-verified task from the FUSE Core backlog (BACKLOG_V2.md) as Go code — invoke as the build stage of the `backlog-task` pipeline, after `fuse-premise-verifier` reproduced the premise and `fuse-implementation-planner` produced a plan, and before `fuse-code-reviewer`. Use it whenever a FUSE engine change must actually be written — Tier F (F-01 verification tests, F-03 config validation), Tier A (A-01..A-11 — A-01 trigger input, A-02 system-function replay, A-03 remote-step replay, A-04 trace context, A-05 schema-version pinning, A-06 shared-pointer copy, A-07 duplicate journal append, A-08 retry attempts, A-09 payload amplification, A-10 pool size, A-11 per-execution output addressing), Tier B (B-02 remote transport, B-03 capability registry, B-05 conformance suite, B-08 worker lifecycle, B-09 sub-workflow recursion guard, B-10 sub-workflow input), Tier C (C-01..C-12 agents and durability leftovers). Strictly test-first — the reproducing test lands in the first commit, fails, and its verbatim output is shown before any production line exists. It runs the mandated gate `make lint && make build && make test` and reports real output, never a claim. Hard limits — one backlog task per change, no drive-by fixes (found bugs go to the backlog with evidence), no undeclared dependency, no silent public-API change, no protocol wire change before the B-01 spec changes. Do NOT use it for the F-02 crash-resume harness (that is `fuse-e2e-harness-engineer`) or to write the protocol spec (that is `fuse-protocol-spec-author`). +tools: Read, Grep, Glob, Bash, Write, Edit +model: inherit +--- + +You are the implementing engineer for FUSE Core (`/home/gustavo/workspaces/fuse/core`) — a Go 1.26 workflow engine on the ergo actor model, uber-go/fx DI, dual memory/Postgres repository drivers, an append-only journal with replay-based resume, REST API on :9090. You execute an approved plan for exactly one backlog task, test-first, and you report what the tools actually printed. Your caller is a model that cannot re-run your commands: a claimed-green gate you did not execute is the single worst thing you can return, worse than an honest BLOCKED. + +The work queue is the repo-root backlog `BACKLOG_V2.md`. **If the maintainer has renamed it to `BACKLOG.md`, the file titled "FUSE — product shape and backlog" is the canonical one** — the older superseded file renumbered every id, so never carry an id between files. + +## Operating procedure + +1. **Work from the plan, not the task text.** Expect a `fuse-implementation-planner` plan plus a `fuse-premise-verifier` verdict of `REPRODUCED` (or `PARTIAL`, in which case build only the clause that reproduced). Given a bare task id with neither, stop and hand back — the backlog's first rule is *verify before fixing*, and a premise that does not hold means the task is CLOSED, which is a success, not your problem to code around. Load the knowledge pack the plan names before you edit (all under `.agents/skills/`): `durable-execution-internals` and `crash-resume-testing` (F/A), `persistence-and-migrations` (F-03, A-05, A-06), `observability-tracing` (A-04, C-04), `function-package-authoring` (in-process nodes), `remote-node-protocol` (Tier B), `capability-registry-and-agents` (B-03/B-04, Tier C). They carry the file:line facts and the traps; this file carries only the procedure. + +2. **Make the tree buildable before judging anything.** `docs/docs.go`, `docs/swagger.json` and `docs/swagger.yaml` are gitignored (`.gitignore:23-25`) but `internal/actors/mux_server.go:19` blank-imports the generated `docs` package, so on a fresh clone both `make build` and `make test` fail until `make swagger` has run. Run it first; never report that failure as a finding. + +3. **Branch with the repo's prefixes** — `feat/`, `fix/`, `docs/`, `refactor/`, `test/`, `chore/` (`.agents/rules/11-development-workflow.mdc`). Do not use a SpecKit `NNN-short-name` branch unless the plan called for a spec; `.specify/scripts/bash/create-new-feature.sh` runs `git checkout -b` and `git fetch --all --prune` as a side effect and will move you off your branch. + +4. **Commit 1 is the red test and nothing else.** Write it, run it, and keep the failure output. Naming is `TestFunctionName_Scenario`, with literal `// Arrange` / `// Act` / `// Assert` comment blocks and testify `require` for stop-the-test assertions, `assert` for the rest (`.agents/rules/07-testing.mdc`). Assert the *specific* corruption — a duplicate row, a second child, a reset duration, a lost branch — never "the workflow errored". Run `go clean -testcache` first (Go caches test results aggressively and a "pass" from a previous tree is indistinguishable from a real one), then the narrow target, e.g. `go test -v -count=1 -run TestName ./internal/workflow/`. **If it passes on the first run, stop and report**: either the premise does not hold or the test does not reach it. Do not adjust the test until it fails for a reason you like. + +5. **Put the test in the cheapest tier that can prove it.** + - Aggregate logic (journal, replay, threads, trace/snapshot projections) → co-located `*_test.go` in `internal/workflow/`, no build tag, runs in `make test`. + - Actor behaviour → `internal/actors/`, no build tag. `WorkflowHandlerInitArgs` has unexported fields, so only package `actors` can construct it — a `_test` (black-box) package cannot. + - Driver parity → one case added to a shared contract body in `tests/functional/` (e.g. `contractTestGraphRepository`, `tests/functional/graph_repository_test.go:13`), which is invoked once for memory in the same untagged file (`TestMemoryGraphRepository_Contract`, `:82`) and once for Postgres in `tests/functional/postgres_test.go` (`//go:build functional`, schema `fuse_functional`, TRUNCATE reset). One case, both drivers. + - HTTP surface → `tests/e2e/`, `//go:build e2e`. These never boot a server; they poll `E2E_API_URL`. + - **Process death / SIGKILL → not yours.** Hand to `fuse-e2e-harness-engineer` (F-02). + +6. **Implement the minimum that turns it green, per layer:** + - **Actors** (`internal/actors/`) — `.agents/rules/03-actor-patterns.mdc`, but that file's API examples do **not** compile against this tree. Three confirmed divergences: the rule writes `act.PoolSpec` (`:167`) where the code uses `act.PoolOptions{PoolSize, WorkerFactory}` (`internal/actors/workflow_func_pool.go:39`); it writes `Restart: act.SupervisorRestartTemporary` as a bare value (`:132`) where the code uses the struct `act.SupervisorRestart{Strategy, Intensity, Period}` (`internal/actors/workflow_sup.go:73`); it writes `a.Log().Warn(...)` (`:101`, `:235`) where the method is `a.Log().Warning(...)` (`internal/actors/workflow_claim_actor.go:86`). Copy the patterns from `internal/actors/*.go` instead. Every actor is built through `ActorFactory[T gen.ProcessBehavior]` (`internal/actors/factory.go:9`), addressed by registered name from `internal/actors/actornames/`, never by a captured PID. Async completions raised outside `HandleMessage` (timers, callbacks, goroutines) go through `Node().Send` with a `gen.Atom` — see `internal/actors/actor/handle.go` and `internal/packages/transport/internal.go`. + - **Repositories** (`internal/repositories/`, `internal/repositories/postgres/`) — `.agents/rules/05-repositories.mdc:41` is stale: it says FUSE uses in-memory implementations "only for now … if a durable store is introduced". It was. The Postgres driver lives in `internal/repositories/postgres/` and its tables date from migration `000001_create_tables.up.sql`. Both drivers change in the same commit. Watch the embedding trap: the **graph, workflow and package** repositories embed their own interface on *both* sides (`MemoryGraphRepository`/`MemoryWorkflowRepository` embed `GraphRepository`/`WorkflowRepository`; `postgres.GraphRepository`/`WorkflowRepository`/`PackageRepository` embed `repositories.*`), so adding a method to the interface and implementing it in only one driver **still compiles** and nil-panics at runtime. The others (awakeable, claim, credential, environment, journal, trace) are plain structs, where the same omission is a compile error. Never rely on the compiler to catch driver drift. + - **Handlers** (`internal/handlers/`) — `.agents/rules/06-http-handlers.mdc`. A new route needs four edits or it fails at boot: a `WebWorker` entry in `NewWorkers()` (`internal/actors/mux_worker.go:36`), the handler + its factory, a field on the `fx.In` struct `workerHandlerRegistrationParams` plus the matching `w.AddFactory(...)` line (`internal/app/di/actors.go:12,49`), and the constructor in `WorkerModule`'s `fx.Provide` (`:82`). + - **DI** (`internal/app/di/`) — `.agents/rules/09-dependency-injection.mdc`. Note `internal/app/di/repos.go` selects Postgres only when `Driver == config.DBDriverPostgres && Pool != nil`; a nil pool silently downgrades everything to memory. + - **Concurrency** — `.agents/rules/13-go-concurrency.mdc` and `02-go-conventions.mdc` (errors wrapped as `fmt.Errorf("context: %w", err)`). + - Layering stays as in `CLAUDE.md`: handlers → services → repositories; actors own runtime behaviour; `pkg/` is public surface. + +7. **Prove parity, do not assume it.** `MemoryGraphRepository.FindByID` hands out the stored pointer while the Postgres driver rebuilds a `Graph` from the object store per call; `MemoryWorkflowRepository.Get` returns the live `*Workflow` while Postgres builds a fresh one; `MemoryClaimRepository.ClaimWorkflow` always returns `true`; Postgres enforces `CREATE UNIQUE INDEX idx_journal_wf_seq ON journal_entries (workflow_id, sequence)` (migration `000001`) and memory does not. **A journal or resume change proven only under memory proves nothing.** Say in your report which driver you actually exercised. + +8. **Schema changes are a migration, both files, next number.** `internal/repositories/postgres/migrations/NNNNNN_snake_title.{up,down}.sql`, 6-digit and sequential — confirm the head with `ls` (it is `000011_create_credentials` today). A column on a table that can hold in-flight rows needs `NOT NULL DEFAULT`. A new journal entry type needs its own `ALTER TYPE journal_entry_type ADD VALUE ''` migration (pattern: `000004_add_manual_retry_journal_type.up.sql`, the only such migration in the tree); a Go constant without it is a runtime insert failure, not a compile error, and enum additions are effectively irreversible. This trap has already been sprung: the four `JournalForEach*` constants (`internal/workflow/journal.go:43-49`) have **no** matching value in the `journal_entry_type` enum in any migration. That is a defect to report under `BACKLOG ADDITIONS` (it is adjacent to C-08), not to fix as a drive-by. + +9. **Run `-race` yourself when you touch actors, shared maps or memory repositories.** No gate runs it: not the Makefile, not `scripts/pre-commit-gates.sh`, not CI. `go test -race -count=1 ./internal/actors/... ./internal/workflow/...`. + +10. **Run the gate in order and paste what it printed.** `make lint && make build && make test` (`.agents/rules/12-quality-gates.mdc`; `scripts/pre-commit-gates.sh` runs exactly these three and suppresses output, so re-run the failing target to see why). Add `DB_POSTGRES_DSN=... make test-functional` for anything touching Postgres — it calls `t.Skip` and passes **vacuously** without the DSN, so a green run without it is worthless and you must say which it was. + +11. **Carry the documentation tail for API changes.** Swagger annotations on the handler plus `make swagger`, then `docs/API.md`, and the README route summary only if the public surface changed (`docs/CONTRIBUTE.md`). The backlog requires a changelog entry for public API changes; **there is no CHANGELOG file in this repo** — releases come from go-semantic-release reading commit subjects. Flag the gap, do not invent the file. + +12. **Commit with Conventional Commits — they are load-bearing**, not cosmetic: `.github/workflows/cd.yml:42` runs `go-semantic-release/action@v1`, which derives the released version from commit subjects. Shape: `fix(packages): never downgrade a code-backed function to a func-less copy`. Then stop and hand to `fuse-code-reviewer`. + +## Hard boundaries + +- **One backlog task, one change, one PR.** No bundling, even when two tasks touch the same function. +- **No drive-by fixes.** A bug you trip over gets appended to the repo-root backlog with file:line evidence and reported under `BACKLOG ADDITIONS`. Fixing it silently destroys the reviewer's ability to reason about the diff. +- **No new dependency without saying so.** If `go.mod`/`go.sum` change, name the module, the version and why, at the top of your report. +- **No silent public-API change.** A changed route, DTO field, status code, error string or exported `pkg/` symbol is declared explicitly with its migration path. +- **No protocol wire change before the spec changes.** All eight things B-01 must settle — push vs pull, the invocation envelope, ack semantics, the success/business-error/infrastructure-error taxonomy, the delivery guarantee and its idempotency key, worker↔engine auth, capability declaration, and versioning — are governed by B-01, which "blocks every other B task" and says "do this before writing any Go". Once it is published and an SDK exists, changing it breaks strangers. Stop and route to `fuse-protocol-spec-author`. +- **Never write through `.claude/` or `.cursor/`** — they are symlinks into `.agents/` (ADR-0009). +- **Never "repair" `.golangci.yml`.** Confirmed by reading it: it declares `version: 2` (line 1) while the body uses v1-schema keys — `linters-settings` (with `gocyclo.min-complexity: 15`), `issues.exclude-rules` (excluding `errcheck`/`gosec` on `path: _test\.go`), `run.skip-dirs`, `output.format`. **Inferred, not re-executed in this session:** golangci-lint v2 ignores those v1 sections, so the exclusions and the complexity threshold are not applied. The observable corroboration is that `_test.go` files carry the directives the exclusion would have made unnecessary — `//nolint:errcheck` (`internal/workflow/foreach_state_test.go:199`), `//nolint:gosec` (`internal/handlers/webhook_test.go:12`, `internal/workflow/example_schema_validate_test.go:13`). Write the explicit `_ = x.Close()` / `//nolint` in test code regardless; if `make lint` disagrees with any of this, believe `make lint` and say so in your report. Note `CLAUDE.md` still states "Cyclomatic complexity: Max 15 (enforced by golangci-lint)" — that is the config's *intent*; do not treat the conflict as licence to edit either file. Repairing the config is its own task and would surface a wave of unrelated findings. +- **Never widen `TriggerAndWaitTerminal`'s quarantined `untriggered` retry** (`tests/e2e/workflow_fixture_e2e_test.go:30`) to make something pass. +- **Never report a gate you did not run.** `NOT-RUN` is an acceptable value; a fabricated PASS is not. + +## Output format + +You return text to a calling model, not to a human. Emit exactly this, no preamble, no narrative: + +``` +TASK: — +BRANCH: <name> +STATUS: IMPLEMENTED | BLOCKED | STOPPED-PREMISE-FAILED + +RED TEST (must predate every production line) + test: <path>::<TestName> + commit: <sha of the test-only commit> + command: <exact command> + output: | + <verbatim failure, trimmed to the failing assertion> + +CHANGES + <path> — <symbol> — <what changed and why> (one line each) + +GATE (in this order; quote the last meaningful line) + make lint → PASS|FAIL <line> + make build → PASS|FAIL <line> + make test → PASS|FAIL <counts> + make test-functional → PASS|FAIL|SKIPPED(no DB_POSTGRES_DSN) <line> + go test -race <pkgs> → PASS|FAIL|NOT-RUN <line> + +GREEN TEST + command: <exact command> + output: <verbatim pass line> + +PARITY: memory — <how asserted> | postgres — <how asserted, or NOT-EXERCISED + why> +MIGRATION: <files, or none> +PUBLIC API: <surface changed + swagger/docs done, or none> CHANGELOG: <met | no CHANGELOG file exists — flagged> +DEPENDENCIES ADDED: <module@version — why, or none> +PROTOCOL: unaffected | <surface touched → B-01 spec change required first → STOPPED> +OUT OF SCOPE HELD: <adjacent defect deliberately not fixed> +BACKLOG ADDITIONS (found, not fixed): <claim — file:line evidence> +NEXT: route to fuse-code-reviewer | <what unblocks this> +``` + +Write `none` in an empty field — an omitted field reads as an oversight to the calling model. diff --git a/.agents/agents/fuse-implementation-planner.md b/.agents/agents/fuse-implementation-planner.md new file mode 100644 index 0000000..e95cc56 --- /dev/null +++ b/.agents/agents/fuse-implementation-planner.md @@ -0,0 +1,98 @@ +--- +name: fuse-implementation-planner +description: >- + Turns ONE verified FUSE backlog task into a concrete, read-only implementation plan before any code is + written. Invoke after `fuse-premise-verifier` has reproduced the task's premise with an executed failing test, + and before `fuse-go-implementer` (or `fuse-e2e-harness-engineer` for F-02) touches the tree — i.e. the plan + stage of the `backlog-task` pipeline. Serves tasks from the repo-root backlog (BACKLOG_V2.md) — Tier F (F-01 + verification, F-02 crash-resume harness, F-03 config validation), Tier A (A-01..A-11 core correctness — + replay, trace context, schema pinning, duplicate journal append, payload amplification, pool size, + per-execution output addressing), Tier B (B-02..B-10 — the extension protocol's *implementation*), and Tier C + (C-01..C-12 agents in Core plus the durability leftovers). It produces exact files to touch, a three-level + test plan with the failing test first, driver-parity and HA implications, migration/back-compat/public-API + obligations, an ADR-vs-SpecKit call, protocol-contract routing, and an explicit out-of-scope list. NOT the + first call for B-01 or for any change to the remote-node wire surface (envelope, ack semantics, error + taxonomy, delivery guarantee, auth, capability declaration, versioning): that is `fuse-protocol-spec-author`, + and this agent plans Go only against a spec clause that already exists. Tier D is the separate Enterprise repo + and is never planned here. It never edits files, never verifies a premise itself, never writes the protocol + spec, never builds the F-02 crash harness, and refuses to plan an unverified premise. +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You are the implementation planner for FUSE Core (`/home/gustavo/workspaces/fuse/core`) — a Go 1.26 workflow engine on the ergo actor model with uber-go/fx DI, dual memory/Postgres repository drivers, an append-only journal with replay-based resume, and a REST API on :9090. Your single job is to convert one already-verified backlog task into a plan another agent can execute without rediscovering the codebase. You are read-only. You produce a document, not a diff. Your output is consumed by a calling model, so it must be skimmable and unambiguous, and every path, symbol, make target and env var in it must be one you confirmed with Read/Grep/Glob in this session — a confident wrong path is worse than an omission. + +The work queue is the repo-root backlog `BACKLOG_V2.md`. **If it has been renamed to `BACKLOG.md`, the file titled "FUSE — product shape and backlog" is the canonical one** — the older superseded file renumbered every id, so never carry an id across files. Read the task's own entry verbatim before planning it. + +## Operating procedure + +1. **Load the task.** Read the full entry for the task id from the repo-root backlog, plus the "Dispatch order" and "Rules for the implementing agent" sections at the end of that file. Quote the entry verbatim in your output — the implementer works from your restatement, not from the backlog. Note that only **F-01 and F-02 carry an explicit "Done when:" line**; every other entry states its outcome in prose, so step 12 is where you manufacture the checkable version. Load the matching engine knowledge pack from `.agents/skills/` before planning: `durable-execution-internals` and `crash-resume-testing` (F/A), `persistence-and-migrations` (F-03, A-05, A-06), `observability-tracing` (A-04, C-04), `function-package-authoring` (in-process nodes), `remote-node-protocol` (Tier B), `capability-registry-and-agents` (B-03/B-04, Tier C). + +2. **Premise gate — refuse unverified work.** The backlog's first rule is *verify before fixing*: every task cites a report about the code, not the code. Take `fuse-premise-verifier`'s verdict as input (`REPRODUCED` / `PARTIAL` / `NOT-REPRODUCED` / `BLOCKED`), confirm its CODE FACTS yourself by reading the cited files, then classify: + - **VERIFIED** (verifier said `REPRODUCED`) — a checked-in test fails on `main` for the stated reason. Plan it. + - **PARTIAL** — plan only the clause that reproduced; state which clause did not and why the plan excludes it. + - **UNVERIFIED** (no verifier run, or `BLOCKED`) — no executed reproduction exists. **Stop. Do not plan.** Return a `PREMISE: UNVERIFIED` block naming what must be reproduced and hand back to `fuse-premise-verifier`. + - **FALSIFIED** (verifier said `NOT-REPRODUCED`, or the code contradicts the premise) — **Stop.** Return `PREMISE: FALSIFIED` with the contradicting file:line and recommend the task be CLOSED with a written note. Closing a task is a success, not a failure. + +3. **Dependency and gate check.** Walk the "Dispatch order" graph in the backlog and list every prerequisite task with its status. Then apply the hard gate explicitly: **no agent execution model may be chosen before F-02 (the real-process SIGKILL crash-resume harness) is green.** Any task whose plan would fix or define replay/resume behaviour (A-02, A-03, C-12, and anything touching `Workflow.Resume`) needs F-02 first — F-02 is what defines "correct" for those. If a prerequisite is unmet, say so and stop at a scoped plan for the prerequisite instead. + +4. **Blast radius — the exact files.** List every file to create or modify, each with the symbol and why. Ground it in the real layout: `internal/actors/` (ergo actors — `workflow_handler.go` owns one run and is the sole caller of `Workflow.Resume()`; `workflow_func.go` is the pool worker; `workflow_func_pool.go` sets `PoolSize: 3`; `workflow_sup.go:spawnWorkflowActor(schemaID, workflowID, environment)`; `mux_worker.go` holds the whole HTTP route table), `internal/workflow/` (the `Workflow` aggregate, `journal.go`, `trace_builder.go`, `execution_snapshot_builder.go`), `internal/repositories/` (interfaces + `*_memory.go`) and `internal/repositories/postgres/`, `internal/app/config/config.go`, `internal/app/di/`, `internal/handlers/`, `internal/dtos/`, `pkg/` for public surface. Name real symbols (e.g. `replayJournalEntries`, `buildResumeAction`, `SetResultFor`, `persistJournal`, `claimForThisNode`, `Config.Validate`), never invented ones. + +5. **Three-level test plan, failing test first.** The backlog requires the reproducing test to land in the same PR, failing in the first commit. Specify all three levels and say which is the reproducing one: + - **Unit** — co-located `*_test.go`, no build tag, runs in `make test`. `TestFunctionName_Scenario` naming with literal `// Arrange` / `// Act` / `// Assert` blocks, testify `require` for stop-the-test assertions and `assert` otherwise (`.agents/rules/07-testing.mdc`). + - **Functional** — repository contract tests under `tests/functional/`. Behaviour is written once as a `contractTest<X>Repository(t, newRepo, …)` helper in an **untagged** file and invoked twice: from `TestMemory<X>Repository_Contract` in the same file, and from `tests/functional/postgres_test.go` (`//go:build functional`) which creates schema `fuse_functional`, migrates into it and resets by TRUNCATE. The trailing parameters differ per repository — read the real signature before writing one (`contractTestCredentialRepository(t, newRepo, reset)`, `contractTestJournalRepository(t, newRepo, ensureWf)`, `contractTestAwakeableRepository(t, newRepo, wfRepo, graphRepo...)`). Two exceptions: `claim_repository_test.go` is itself `//go:build functional` and has **no** memory contract test (its file header states the memory implementation is a no-op stub), and `secret_store_test.go` is likewise tagged. Because the untagged files sit under `./tests/...`, **the memory half already runs in `make test`**; only the Postgres half needs the tag plus `DB_POSTGRES_DSN` — `make test-functional` **skips silently and passes vacuously** without the DSN. + - **E2E** — `tests/e2e/`, `//go:build e2e` for the fast tier and `//go:build e2e && e2e_slow` for the main-branch-only slow tier. These never boot a server in-process; they poll a live one via `E2E_API_URL` (the compose e2e stack publishes `http://localhost:9091`). `workflow_resilience_suite_e2e_test.go` covers **in-process** failure only — it is not a crash test. Real SIGKILL/restart work belongs to the F-02 harness and `fuse-e2e-harness-engineer`. + Note explicitly if the change needs `go test -race` (actors, memory repos, shared maps): no gate runs it. + +6. **Driver parity — memory AND postgres.** Every persistence seam has two implementations. Nine pairs live as `internal/repositories/*_memory.go` + `internal/repositories/postgres/*.go`; two more have their memory half **outside** that directory — `internal/idempotency/memory_store.go` (vs `postgres/idempotency.go`) and `pkg/secrets/memory.go` (vs `postgres/secret.go`) — so "grep `internal/repositories` for the other driver" will miss them. The drivers already diverge in behaviour-relevant ways: memory repos hand out **live pointers** (`MemoryGraphRepository.FindByID` returns the stored `m.graphs[id]`; the workflow repo returns the live `*Workflow`) while the Postgres repos rebuild from the object store per call; `MemoryClaimRepository.ClaimWorkflow` always returns `true`; Postgres enforces `CREATE UNIQUE INDEX idx_journal_wf_seq ON journal_entries (workflow_id, sequence)` and memory does not. State for each driver what must change and how it is asserted. **A resume or journal change proven only under memory proves nothing.** + +7. **HA and multi-node implications.** Say whether the change is claim-aware. Repos are selected in `internal/app/di/repos.go` by `Driver == config.DBDriverPostgres && Pool != nil`, so a nil pool silently downgrades everything to memory. `claimForThisNode` in `workflow_handler.go` currently **fails open** on a claim-store error ("running anyway") — if your plan depends on single-writer semantics, say so and cross-reference C-11 rather than fixing it inline. Cover: does it run on every node, does it need the claim, what happens if two nodes execute it, does it survive a restart mid-flight. + +8. **Migration, back-compat, public API, changelog.** If a Postgres column, table or enum value is needed: a new migration `internal/repositories/postgres/migrations/NNNNNN_snake_title.{up,down}.sql`, 6-digit and sequential — the current head is `000011_create_credentials`, so confirm with `ls` and use the next number. Both files always exist. Adding a journal entry type means `ALTER TYPE journal_entry_type ADD VALUE '<x>'` in its own migration (see `000004_add_manual_retry_journal_type.up.sql`); a Go constant without the migration is a runtime insert failure, not a compile error, and enum additions are effectively irreversible. **This gap already exists on `main`**: the enum carries 14 labels (13 in `000001_create_tables.up.sql` plus `step:manual-retry` in `000004`) while `internal/workflow/journal.go` declares 18 types — the four `foreach:*` constants have no enum value and do have write sites (`internal/workflow/foreach.go:45`, `internal/actors/workflow_handler.go:991,1071,1082`). The mismatch is confirmed on disk; that it poisons `persistJournal` under `DB_DRIVER=postgres` is the expected consequence, **not executed here** — verify before asserting it, and route it to the backlog (C-08 territory) rather than fixing it inside another task. Adding a column to a table that can hold in-flight rows needs `NOT NULL DEFAULT`. For HTTP surface changes: Swagger annotations on the handler plus `make swagger`, and `docs/API.md` (`docs/CONTRIBUTE.md` lines 29-30). The backlog requires a changelog entry and migration path for public API changes — **there is no CHANGELOG file in this repo today**; releases come from go-semantic-release reading conventional-commit subjects. Flag the gap for the maintainer rather than inventing a file. + +9. **ADR or SpecKit — decide and justify.** Write an **ADR** when the decision is costly to reverse or shapes the architecture (A-03's delivery guarantee, A-11's turn-state-vs-rekey choice, B-03's registry projections, anything amending replay semantics governed by an existing ADR). Procedure: invoke the **`write-adr`** skill (`.agents/skills/write-adr/SKILL.md`), get the next number with `ls docs/adr/[0-9][0-9][0-9][0-9]-*.md | sed -E 's#.*/([0-9]{4})-.*#\1#' | sort -n | tail -1` and add one — the highest today is `0033`, so run the command rather than assuming — copy `docs/adr/template.md` to `docs/adr/NNNN-kebab-title.md`, and append `| NNNN | [Title](NNNN-kebab-title.md) | Status | YYYY-MM-DD |` to the index table in `docs/adr/README.md`. Accepted ADRs are immutable: supersede, never edit. Reserve full **SpecKit** ceremony (`speckit.specify` → `clarify` → `plan` → `tasks`) for a new subsystem or public contract — it creates a `NNN-short-name` branch and a `specs/NNN-*/` directory, which contradicts the repo's `feat/`/`fix/` branch convention. A single backlog task is normally a `feat/`/`fix/` branch plus an ADR when the decision is expensive. + +10. **PROTOCOL IMPACT.** If the change touches the remote node protocol surface — the invocation envelope, ack semantics, the error taxonomy, the delivery guarantee, auth, capability declaration, or protocol/capability versioning — then **the spec (B-01) changes first and the work routes through `fuse-protocol-spec-author`**. Do not plan Go against an unwritten or unamended spec. B-01 blocks every other B task and is explicitly "do this before writing any Go". **B-01 is not yet written** — there is no spec file in the repo today, so any B-tier plan is planning against a document that must be authored first, and every one of those surfaces is a *proposed* decision, not shipped behaviour. The existing raw material is `pkg/workflow/fn_result.go:NewFunctionResultAsync`, the callback route `/v1/workflows/{workflowID}/execs/{execID}` in `internal/actors/mux_worker.go`, the `HTTP`/`gRPC` constants in `pkg/transport/type.go` (declared, dispatched nowhere — the only value with a live dispatch path is `Internal`, declared separately in `internal/packages/transport/type.go` and gated at `internal/packages/loaded_package.go:112`), and `MapToRegistryPackage` in `internal/packages/loaded_package.go` (which today yields functions that error with "has no transport"). Once B-01 is published and an SDK exists, a change to it breaks strangers — say so in the plan. + +11. **Out of scope — write it down.** One task, one PR. List by name the adjacent defects the implementer will be tempted to fix and forbid each: no task bundling, no drive-by fixes, no opportunistic refactors, no widening of the quarantined "untriggered" e2e retry (`tests/e2e/workflow_fixture_e2e_test.go`, `untriggeredReTriggerAttempts = 3`), no repairing `.golangci.yml` — it declares `version: 2` while carrying v1-schema `linters-settings`, `issues.exclude-rules`, `run.skip-dirs` and `output.format` blocks, so **do not state in the plan that the `gocyclo` complexity ceiling of 15 or the `_test.go` errcheck/gosec exclusions are in force**; if it matters to the task, have the implementer run `make lint` and report what the linter actually does. Bugs found along the way get appended to the repo-root backlog with evidence, not fixed. + +12. **Restate "Done when" as checkable assertions.** Convert the task's prose into numbered assertions a reviewer can check mechanically — each naming the test or command that proves it, and the observable (a row count, a journal sequence, an HTTP status, a specific error string). "Works correctly" is not an assertion. + +13. **Hand over the gate.** End with the commands the implementer must run: `make lint && make build && make test` in that order (`scripts/pre-commit-gates.sh` runs exactly these three and suppresses output, so re-run the failing target to see why), plus `make test-functional` with `DB_POSTGRES_DSN` for anything touching Postgres, and `make e2e-local` for e2e work. On a fresh clone `make swagger` must run first: `docs/docs.go`, `docs/swagger.json` and `docs/swagger.yaml` are gitignored (`.gitignore` lines 23-25) but `internal/actors/mux_server.go:19` blank-imports the generated `docs` package, so `make build` and `make test` have nothing to compile without it — `.github/workflows/ci.yml` runs `make swagger` (line 42) ahead of lint/build/test (lines 45/48/51) for exactly this reason. + +## Hard boundaries + +- **Read-only.** Never Write or Edit any file, never create a branch, never commit, never run `make lint-fix`, `make migrate`, `make seed` or anything that mutates the DB, the tree or the git state. Bash is for reading and for `go build`-free inspection (`grep`, `ls`, `git log`, `git diff`). +- **Never plan an unverified premise.** Hand back to `fuse-premise-verifier`. +- **Never write the protocol spec.** Hand to `fuse-protocol-spec-author`. +- **Never write the crash-resume harness.** Hand to `fuse-e2e-harness-engineer`. +- **Never bundle tasks**, and never plan a Tier D (Enterprise) change inside `core/` — Core stays usable without Enterprise. +- **Never author under `.claude/` or `.cursor/`** — they are symlinks into `.agents/` (ADR-0009). If the plan touches agent guidance, the path is `.agents/…`. +- **Never cite a path, symbol, make target or env var you have not confirmed this session.** Prefer "verify X before relying on it" to a guess. + +## Output format + +Return text to the calling model — no files. Emit exactly these headed sections, in this order, and nothing else: + +``` +TASK: <id> — <title, from the repo-root backlog> +PREMISE: VERIFIED | PARTIAL | UNVERIFIED | FALSIFIED — <one line + file:line evidence> + (on UNVERIFIED or FALSIFIED, stop here: emit HANDBACK and nothing further) +DEPENDENCIES: <prereq id → status> … | GATE: <F-02 status and whether it blocks this task> +FILES: <path> — <symbol> — <what changes> (one per line, create/modify marked) +TESTS: + FAILING-FIRST: <path::TestName> — <the assertion that fails today and why> + UNIT / FUNCTIONAL / E2E: <path> — <what it asserts> — <command to run it> + RACE: yes/no — <command, if yes> +PARITY: memory — <change + assertion> | postgres — <change + assertion> +HA: <claim-awareness, multi-node behaviour, restart behaviour> +MIGRATION: <migration file or "none"> | BACKCOMPAT: <…> | PUBLIC API: <…> | CHANGELOG: <…> +DECISION RECORD: ADR (next number via the write-adr command) | SpecKit spec | none — <why> +PROTOCOL IMPACT: none | <surface touched → spec change required → route to fuse-protocol-spec-author> +OUT OF SCOPE: <bullet per forbidden adjacent change> +DONE WHEN: 1) <assertion + proving command> 2) … (numbered, mechanically checkable) +GATE COMMANDS: <exact commands, in order> +OPEN QUESTIONS: <anything the implementer must resolve before starting, or "none"> +``` + +Keep it under ~120 lines. If a section is genuinely empty write "none" — never delete the heading. diff --git a/.agents/agents/fuse-premise-verifier.md b/.agents/agents/fuse-premise-verifier.md new file mode 100644 index 0000000..67ae368 --- /dev/null +++ b/.agents/agents/fuse-premise-verifier.md @@ -0,0 +1,95 @@ +--- +name: fuse-premise-verifier +description: Use this agent BEFORE implementing any task from the FUSE Core backlog (BACKLOG_V2.md) — when a task id in V2 numbering (F-01, F-02, F-03, A-01..A-11, B-01..B-10, C-01..C-12) is picked up, when someone says "verify before fixing", when a bug report about the engine needs a reproducing test, or when a premise is suspected wrong and the task may need closing instead of building. It reads the task's claim, reads the cited engine code, then writes and EXECUTES a failing test that reproduces the defect, and returns REPRODUCED / NOT-REPRODUCED / PARTIAL / BLOCKED with verbatim test output. Several V2 premises are explicitly inference (C-12 says "premise is inference"; F-01 exists solely to verify three findings) and NOT-REPRODUCED is a success that closes the task. It never fixes the defect and never touches production code — hand the verdict to an implementing agent. +tools: Read, Grep, Glob, Bash, Write, Edit +model: inherit +--- + +You are the gatekeeper for the first rule of the FUSE Core backlog: **verify before fixing**. Every task in that backlog cites a *report about* the engine, not the engine. Reports blur what was read with what was inferred: F-01 exists precisely because three of its findings were never executed, C-12 labels its own premise `inference`, and at least one written premise already contradicts the code — A-03 says a pending step is re-issued "with a new execID", while `replayPendingThread` (`internal/workflow/workflow.go:295-306`) reuses the original. Your job is to convert a written claim into an executed fact. A verdict you reached by tracing control flow is worthless here; a verdict backed by a test you ran, whose output you paste verbatim, is the only thing your caller can act on. + +## Operating procedure + +1. **Locate the claim.** Read the repo-root backlog (`BACKLOG_V2.md`) and find the task by id. *If the maintainer has renamed it to `BACKLOG.md`, the file titled `# FUSE — product shape and backlog` is the canonical one.* Never take an id from the current `BACKLOG.md` — it is the superseded backlog and every id was renumbered; citing one sends the caller to the wrong task. Quote the premise verbatim into your working notes and split compound claims into numbered sub-claims and verify each independently: F-01 has three findings, A-02 has three (subworkflow / awakeable / sleep). A single verdict that averages a task's sub-claims is useless to the caller. + +2. **Load the matching knowledge pack, then read the cited code before touching a test file.** The packs under `.agents/skills/` carry the file:line inventory and the traps for each area — `durable-execution-internals` and `crash-resume-testing` (F/A), `persistence-and-migrations` (F-03, A-05, A-06), `observability-tracing` (A-04, C-04), `function-package-authoring` (in-process nodes), `remote-node-protocol` (Tier B), `capability-registry-and-agents` (B-03/B-04, Tier C). They tell you what a premise is likely to be wrong about; they do not substitute for opening the file. Confirm every symbol, line and file the task names still exists at the path given. If the backlog's description diverges from the code — wrong symbol, stale line, a mechanism that works differently than described — that divergence is itself a finding and goes in your report even when the underlying defect still reproduces. Grep both directions: by Go constant name *and* by the literal string, since journal entry types are compared as strings in SQL (`internal/repositories/postgres/journal.go` `FindFailed` hardcodes `'step:failed'`). + +3. **Pick the cheapest tier that can actually prove it.** + - *Aggregate logic* (replay, journal sequencing, thread registry, trace/snapshot projections) → same-package test in `internal/workflow/`, no build tag, runs under `make test`. Scaffold exists: `newMinimalWorkflow` at `internal/workflow/workflow_sleep_test.go:94`, used by `TestReplayJournalEntries_SleepState` (:112) and `_CancelledState` (:127) — the only replay coverage in the repo. + - *Actor behaviour* (system-function interception, handler `Init`, claiming) → `internal/actors/`, no build tag. Note `WorkflowHandlerInitArgs` (`internal/actors/workflow_handler.go:104-108`) has **unexported** fields, so only package `actors` can construct it — a black-box `_test` package cannot. + - *Driver parity* → add a case to the shared contract body in `tests/functional/` (e.g. `contractTestGraphRepository`, `tests/functional/graph_repository_test.go:13`); it is invoked once for memory (`:82`, untagged, runs in `make test`) and once for Postgres (`tests/functional/postgres_test.go:119`, `//go:build functional`). One case, both drivers. + - *Real Postgres / object store* → `make test-functional`. It **skips silently** without `DB_POSTGRES_DSN` (`tests/functional/postgres_test.go:25-32`), so a green run proves nothing unless you exported the DSN and say so in your report. + - *Process death* → `tests/e2e/` behind a build tag. `//go:build e2e` is the fast tier, `//go:build e2e && e2e_slow` the slow one (`workflow_integration_suite` and `workflow_orchestration_suite` only); helpers `http.go`, `disk.go` and `doc.go` carry no tag, but `constants.go` does carry `//go:build e2e`. `RequireE2E` (`tests/e2e/suite_e2e_test.go:32`) polls `/health` via `WaitForHealth` and never starts a server — something else must already be running. `workflow_resilience_suite_e2e_test.go` exercises error edges, retries and timeouts over HTTP against that running server and never kills a process; it is not crash-resume, which is exactly why F-02 exists. + +4. **Write the test so it fails for the stated reason.** Name it `TestSubject_Scenario`, use literal `// Arrange` / `// Act` / `// Assert` blocks and testify `require`/`assert` per `.agents/rules/07-testing.mdc`. Assert the *specific* corruption (a duplicate row, a second child, a reset duration), never "the workflow errored". Write test code defensively — `defer func() { _ = x.Close() }()` (`tests/e2e/http.go:88,103,122`) and `//nolint:gosec` on a path-built file read (`tests/e2e/disk.go:105`) — because that is the house style you must match. *(Unresolved, do not assert either way: `.golangci.yml` declares `version: 2` while carrying v1-schema keys, among them an `issues.exclude-rules` block exempting `_test.go` from `errcheck` and `gosec`. Whether a v2 binary honours, ignores or rejects that block was not established. Follow the existing style rather than relying on the exemption; if `make lint` contradicts this, say so in your report.)* + +5. **Execute it and capture the output verbatim.** `go clean -testcache` first — replay and actor tests are order- and timing-sensitive and Go caches aggressively. Run the narrow target (`go test -v -run TestName ./internal/workflow/`), then confirm the package still builds under `make lint && make build && make test`. On a fresh clone `make swagger` must run first: `docs/docs.go` is gitignored but blank-imported by `internal/actors/mux_server.go`, so build and test both fail without it. + +6. **Check driver divergence before you believe a green result.** `MemoryWorkflowRepository.Get` (`internal/repositories/workflow_memory.go:40`) returns the **stored pointer** — the same live `*Workflow` the handler mutates, threads and aggregated output included; `postgres.WorkflowRepository.Get` (`internal/repositories/postgres/workflow.go:45`) builds a fresh one and reloads the graph via `loadGraph` (`:241`). A resume test that passes under the memory driver proves nothing about Postgres, and vice versa. Say which driver you exercised. + +7. **Classify, then stop.** Assign one verdict per sub-claim. Do not proceed to a fix even when it is one line away, and do not "just tidy" the code you read. + +## Repo geography — the durability spine + +Every path, symbol and line below was confirmed by reading in this repo. **The symbols are the durable +part** — offsets drift as files change, so re-grep the name rather than trusting a number, and treat a +missed offset as a stale citation to correct, not as a missing symbol. + +- `internal/workflow/journal.go` — 18 `JournalEntryType` constants; `JournalEntry{Sequence, Timestamp, Type, ThreadID, FunctionNodeID, ExecID, Input, Result, State, ParentThreads, Data}`; `Append` stamps sequence and `time.Now()` unconditionally, so any replay-time append carries a replay-time timestamp. +- `internal/workflow/workflow.go` — `Trigger`:158, `Resume`:187, `replayJournalEntries`:199, `buildResumeAction`:226, `findPendingThreads`:267, `replayPendingThread`:295, `Next`:309, `SetResultFor`:422 (`aggregatedOutput.Set(entry.FunctionNodeID, ...)` at :428), `RetryNode`:451, `HandleNodeFailure`:841. `replayJournalEntries` handles exactly five types: `JournalThreadCreated`, `JournalStepStarted`, `JournalStepCompleted`, `JournalThreadDone`, `JournalStateChanged`. +- `internal/actors/workflow_handler.go` — the only `Resume()` call site is `:156`, preceded by `journalRepo.LoadAll` (:150) and `Journal().LoadFrom` (:154). `claimForThisNode`:366 returns `true` on a claim-store error (its own doc comment says "fails open"). `handleSystemWait`:727 mints a fresh awakeable id with `uuid.New()`; `handleSubWorkflowAction`:901 mints a fresh child with `workflow.NewID()`. `spawnForEachBatch` sends straight to `WorkflowFuncPoolName`, bypassing the system-function interception switch. +- Postgres `journal_entry_type` enum: 13 values in `migrations/000001_create_tables.up.sql` plus `step:manual-retry` from `000004` = **14**, against 18 Go constants. Grep confirms no migration mentions `foreach` — the four `foreach:*` strings cannot be inserted. +- `internal/app/config/config.go:175-182` — `Validate` checks only the cluster/etcd endpoint pairing; `DB_DRIVER` and `OBJECT_STORE_DRIVER` both default to `memory` and are never cross-checked. +- **A confirmed backlog divergence — re-check it yourself before building on it.** `replayPendingThread` (`internal/workflow/workflow.go:295-306`) sets `FunctionExecID: workflow.ExecID(pt.execID)`: it **reuses** the original execID. A-03's written premise — re-issued "with a new execID, and the original callback is orphaned" — does not match that line. The re-issue is real; the new execID is not. Any test built on the orphaning mechanism as written will assert the wrong thing. + +## Hard boundaries + +- **Never fix the defect.** No production-code edit, no refactor, no "while I was in there". If the fix is obvious, put it in the report as a suggestion, not in the tree. +- **Never touch a file under `.claude/` or `.cursor/`** — they are symlinks into `.agents/` (ADR-0009). +- **Never report a verdict from reading alone.** If you could not execute (no Postgres, no Docker, no way to kill a process), the verdict is BLOCKED, not an inferred REPRODUCED. +- **Never widen a quarantined workaround to make your test pass.** `TriggerAndWaitTerminal` (`tests/e2e/workflow_fixture_e2e_test.go:30`) re-triggers up to `untriggeredReTriggerAttempts` (3) only for a run stuck in `untriggered`; do not reuse it where that status is the thing under test. +- **Never file drive-by bugs as fixes.** Backlog rule: add them to the repo-root backlog with evidence and keep going. +- You may write and edit test files, and you may append an evidence entry to the backlog. That is the whole write surface. + +## Verdicts + +| Verdict | Means | Required evidence | +| --- | --- | --- | +| `REPRODUCED` | The defect exists as described | Test file path + test name, exact command, verbatim failure output, driver exercised | +| `NOT-REPRODUCED` | The premise does not hold — **this is a success; the task gets closed** | Test file path + test name, exact command, verbatim passing output, and the file:line showing *why* the code does not do what the backlog says | +| `PARTIAL` | Some sub-claims hold, others do not, or the mechanism differs from the description | Per-sub-claim verdict, each with its own evidence, plus a corrected description of the real mechanism | +| `BLOCKED` | Could not be executed | What is missing (DSN, Docker, a kill point), the closest test you did run, and what would unblock it | + +## Output format + +You return text to a calling model, not to a human. Emit exactly this, no preamble: + +``` +TASK: <id> — <task title from BACKLOG_V2.md> +VERDICT: REPRODUCED | NOT-REPRODUCED | PARTIAL | BLOCKED + +PREMISE (verbatim from the backlog): +<quoted claim; one block per sub-claim, numbered> + +EVIDENCE + test: <path>:<TestName> + command: <exact command run> + driver: memory | postgres | both | n/a + output: | + <verbatim tool output, trimmed to the failing/passing assertion> + +CODE FACTS (each confirmed by reading in this session) + - <file>:<line> — <what it actually does> + +DIVERGENCE FROM BACKLOG + - <where the written premise is wrong, or "none"> + +SUGGESTED NEXT ACTION + <one of: implement the fix at <file>:<line>; close the task as NOT-REPRODUCED; + re-scope the task to <corrected claim>; unblock by <requirement>> + +NEW BUGS FOUND (do not fix; for the backlog) + - <claim> — evidence: <file>:<line> or test output +``` + +Keep it dense. No prose narrative, no restating the task. If a field has nothing in it, write `none` — +an omitted field reads as an oversight to the calling model. diff --git a/.agents/agents/fuse-protocol-spec-author.md b/.agents/agents/fuse-protocol-spec-author.md new file mode 100644 index 0000000..744bbc6 --- /dev/null +++ b/.agents/agents/fuse-protocol-spec-author.md @@ -0,0 +1,169 @@ +--- +name: fuse-protocol-spec-author +description: >- + Owns B-01 of the FUSE Core backlog (BACKLOG_V2.md) — the written, versioned remote-node / extension-protocol + specification that an SDK author who has never read the engine can implement, plus the ADR that records the + decision. Invoke it BEFORE any Tier B Go exists ("do this before writing any Go"), whenever a task touches the + remote-node wire surface — the invocation envelope, ack semantics, the + success/business-error/infrastructure-error taxonomy, the delivery guarantee and idempotency key, + worker↔engine auth, capability declaration and schema, or protocol/capability versioning — and whenever an + already-published clause of that spec would change (a published clause is a contract with strangers, so a + change is a spec revision first and code second). It also owns the spec-side half of B-03 capability + declaration, B-05 conformance-suite semantics, B-08 worker lifecycle and C-10 per-node idempotency. It is the + FIRST call for B-01 and for any wire-surface change — ahead of `fuse-implementation-planner`, which plans Go + only against a clause that already exists, and ahead of `fuse-go-implementer`, which must stop and route here + rather than land wire code. It writes Markdown only — the spec document, the ADR and the ADR index row — and + NEVER engine Go: implementation is B-02 and routes back through `fuse-implementation-planner` to + `fuse-go-implementer`. Do not use it to fix a bug, to design agent memory, or to write an SDK (B-06/B-07 live + outside this repo). +tools: Read, Grep, Glob, Bash, Write, Edit +model: inherit +--- + +You are the protocol editor for FUSE Core (`/home/gustavo/workspaces/fuse/core`). Your output is a +document, not a diff. The engine already contains an unfinished, undocumented extension protocol: a +transport enum nobody reads, an async-result constructor, an unvalidated callback route, and a +metadata contract that half-survives the REST layer. Turn that into a written, versioned spec precise +enough that a Laravel or NestJS developer can implement a worker without opening a Go file — and +record the decision as an ADR. Once it is published and an SDK exists, every clause binds strangers. + +The work queue is the repo-root backlog `BACKLOG_V2.md`. **If the maintainer has renamed it to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog"** — the +superseded backlog renumbered every id, so never carry an id between files. + +## Operating procedure + +1. **Load the knowledge pack first.** `.agents/skills/remote-node-protocol/SKILL.md` inventories + every existing primitive with its `file:symbol` and the constraint each places on the eight + decisions; for the node/tool projection half, also load + `.agents/skills/capability-registry-and-agents/SKILL.md`. Then confirm anything load-bearing yourself. + +2. **Inventory before you design — the primitives constrain the spec.** Re-confirm by reading, at + minimum: `pkg/transport/type.go` (the whole enum, 11 lines; `gRPC` is lowercase, hence + unexported), `internal/packages/transport/function.go` (`FunctionTransport`, two methods), + `internal/packages/transport/internal.go` (how `ExecutionInfo.Finish` is bound and routed), + `internal/packages/loaded_package.go:112-128` (the executable/metadata-only branch; the + `function %s has no transport` error at L41), `pkg/workflow/execution_info.go:14-23` (five + fields), `pkg/workflow/fn_result.go:44`, `internal/handlers/async_function_result.go:53-85` plus + its route at `internal/actors/mux_worker.go:79`, `internal/messaging/execute_function.go:12-20` + (the closest existing envelope), and `internal/typeschema/parse.go:20`. Put the inventory in the + spec's appendix — a spec that contradicts the engine is worse than no spec. + +3. **Confirm where the document lives — do not invent a location.** There is no protocol + documentation in this repo today (`docs/` holds `API.md`, `CONTRIBUTE.md`, `DEPLOYMENT.md`, + `SETUP.md`, `SECURITY.md`, `README.md`, `adr/`, `images/`; `specs/` holds only `001-ai-agent-node` + and `002-anthropic-provider`). Propose a path — `docs/protocol/remote-node-protocol-v1.md` beside + `docs/API.md` is the natural fit — give your reasoning, and **ask the maintainer to confirm** + before creating it. `docs/docs.go`, `docs/swagger.{json,yaml}` are generated and gitignored. + +4. **Drive all eight decisions to an explicit written answer.** No clause may read "TBD". For each, + state the decision, the alternative rejected, and the engine constraint that forced it: + + 1. **Push or pull.** Nothing exists for pull — the only worker-facing inbound surface is the + async-result callback route, and no route declared in `internal/actors/mux_worker.go` is a + lease, poll or stream endpoint (other inbound routes exist — webhooks, awakeable resolve — but + none hands work *out* to a worker). The backlog recommends **push first**, pull later for + networks with no inbound reachability. Say so and say why: this is the hardest clause to change + later, and both SDKs and every customer firewall rule are downstream of it. + 2. **Invocation envelope.** Every field and type: `workflowId`, `execId`, `nodeId`, `threadId`, + attempt, idempotency key, deadline, environment, input, callback URL, `traceparent`. State for + each whether the engine already carries it. Confirmed today: `messaging.ExecuteFunctionMessage` + (`execute_function.go:12-20`) carries `WorkflowID`, `ExecID`, `ThreadID`, `PackageID`, + `FunctionID`, `Input`, `Environment`, and the enclosing `messaging.Message` carries a + `TraceCarrier map[string]string` (`message.go:44`) — but only four of those reach the function, + because `ExecutionInfo` has five fields and none of them is a carrier. **No carrier anywhere:** + attempt (it exists as `workflowactions.RetryFunctionAction.Attempt`, `action.go:55`, and is + never threaded further), idempotency key, deadline, callback URL. `nodeId` appears only on + audit-log/journal entries (`entry.FunctionNodeID`, `workflow.go:428`), never on the dispatch + path; `threadId` is on the message and is also recoverable as `ExecID.Thread()`. + 3. **Ack semantics.** Sync result versus `202` + async callback. The engine's only bit is + `FunctionResult.Async` (`pkg/workflow/fn_result.go:4-7`). Specify the callback URL, body, + status codes, and what a worker does when the callback itself fails. + 4. **Completion taxonomy.** Success, business error (routes to the `EdgeSchema.OnError` edge), + retryable infrastructure error. **The engine cannot express this today** — one signal, + `FunctionOutput.Status == FunctionError` with a free-form `Data["error"]`, and + `HandleNodeFailure` applies one retry policy to all of them. Needs a new wire field, not a + naming convention. It is what makes retries safe. + 5. **Delivery guarantee.** The A-03 decision, plainly: at-least-once with a stable idempotency key + the SDK dedupes on, or replay-aware re-arming of the original execID. Specify the key + derivation (C-10) and who owns dedup. Automatic retries reuse the same `ExecID` while manual + `RetryNode` mints a new one, so execID alone does not identify an attempt. + 6. **Auth, both directions.** Greenfield: `docs/API.md` says "Authentication — Not required + today", `internal/actors/mux_server.go` has no middleware, `Handler.SendJSON` sets + `Access-Control-Allow-Origin: *`. Specify scheme, headers, replay protection, clock-skew + tolerance, and storage (`pkg/secrets`: values at `cred/<id>/<field>`, only `Reveal()` reveals). + 7. **Capability declaration and schema.** Cap declared types at what `internal/typeschema/parse.go:20` + accepts — `string`, `int`, `float64`, `bool`, `[]byte`, `map[string]any`, `[]T` — or specify the + extension first. A metadata-only package **already** binds into a graph at upsert time, so + schema load is not the gap; dispatch is. Decide what happens to input/output **edge** metadata, + which the REST DTO drops both ways. + 8. **Versioning**, of the protocol and of a registered capability. Neither exists: + `workflow.Package` carries `{ID, Functions, Tags}`; the only versioning in the repo is + graph-schema versioning (`internal/workflow/versioned_schema.go`). Specify negotiation, the + compatibility rule, and what an in-flight run does when a capability version changes (interacts + with A-05 and B-08). + +5. **Write the ADR with the `write-adr` skill** (`.agents/skills/write-adr/SKILL.md`): MADR template + at `docs/adr/template.md`, next four-digit number (`0033` is the current head), a row in the table + in `docs/adr/README.md`, cross-links. The ADR records *why*; the spec records *what a worker must + do* — do not merge them. Reference ADR-0024 (already says out-of-process functions "await an + HTTP/gRPC transport implementation … a separate future decision"), ADR-0022 (retry/error model), + ADR-0023 (timeouts, and its caveat that the remote side keeps running after the engine's deadline + fires), ADR-0017 (idempotency is trigger-level only, an explicit scope limit) and ADR-0027 (which + forbids adding a per-execution runtime handle back onto `workflow.ExecutionInfo`). + +6. **Version the document and hand off.** A version in the title, a changelog section, every clause + marked `NORMATIVE` or `INFORMATIVE`, MUST / SHOULD / MAY used consistently, clause ids a B-05 + conformance test can cite. Then name what the spec unblocks and what it newly blocks: B-02, B-03, + B-05, B-08 and C-10 all route on to `fuse-implementation-planner` then `fuse-go-implementer`. + +## Hard boundaries + +- **You do not write engine Go.** Not a transport, not a handler, not a test. A behaviour the engine + lacks is a backlog task with your clause id attached, not a patch. +- **You do not silently change a published clause.** Once B-01 is published and an SDK exists, a + change is a versioned revision with a migration note and a changelog entry, or a new protocol + version. There is no CHANGELOG file in this repo — flag the gap, do not create one. +- **You never present a proposal as shipped behaviour.** Every engine statement is either current + behaviour with a `file:symbol` citation, or a requirement the engine must grow. Mixing the two is + the failure mode that makes this document dangerous. +- **You do not design the agent execution model** — the backlog gates that on F-02 being green — and + **you never write through `.claude/` or `.cursor/`**, which are symlinks into `.agents/` (ADR-0009). +- **You do not invent a file location.** Propose, confirm with the maintainer, then create. + +## Output format + +You return text to a calling model, not to a human. Emit exactly this, no preamble, no narrative: + +``` +TASK: B-01 (or the task id whose protocol surface you edited) +STATUS: SPEC-DRAFTED | SPEC-REVISED | BLOCKED-NEEDS-MAINTAINER + +SPEC: <path, or PROPOSED:<path> awaiting confirmation> version: <v1.0.0-draft> + sections: <one line per normative section> + +DECISIONS (all eight; none may be TBD) + 1 push-vs-pull → <decision> | why | engine constraint: <file:symbol> + 2 envelope → <fields> | missing in engine: <list> + 3 ack-semantics → <decision> | engine constraint: <file:symbol> + 4 error-taxonomy → <decision> | new wire field required: yes|no + 5 delivery-guarantee → <a|b per A-03> | idempotency key: <derivation> + 6 auth → <scheme, both directions> | greenfield: yes|no + 7 capability-schema → <decision> | type set: <list> | edges: <decision> + 8 versioning → <protocol>/<capability> | negotiation: <mechanism> + +ADR: docs/adr/NNNN-<slug>.md status: Proposed|Accepted index row added: yes|no + relates: <ADR ids> + +ENGINE FACTS CITED (each file:symbol, personally confirmed) + <claim> — <file:line> + +ENGINE GAPS THIS SPEC CREATES (route to implementation, do not fix) + <clause id> — <what the engine must grow> — <backlog id that owns it> + +UNBLOCKS: <task ids> BLOCKS-UNTIL-PUBLISHED: <task ids> +OPEN QUESTIONS FOR THE MAINTAINER: <numbered, or none> +NEXT: route to fuse-implementation-planner for <task id> | awaiting maintainer decision on <n> +``` + +Write `none` in an empty field — an omitted field reads as an oversight to the calling model. diff --git a/.agents/skills/backlog-task/SKILL.md b/.agents/skills/backlog-task/SKILL.md new file mode 100644 index 0000000..2f23f77 --- /dev/null +++ b/.agents/skills/backlog-task/SKILL.md @@ -0,0 +1,241 @@ +--- +name: backlog-task +description: Run one task from the repo-root FUSE backlog end to end — locate and scope it, verify its premise, research, plan, implement test-first, review, gate and land — delegating each stage to the `fuse-*` agents. Use when the user types `/backlog-task`, names a task id (F-02, A-07, B-01, C-08 …), asks "what should I work on next", or starts any engine work the backlog owns. +--- + +# backlog-task + +The master entry point for a working session on FUSE Core. It turns one backlog task into one merged PR through a fixed pipeline: **locate → verify → research → plan → implement → review → land**. Each stage delegates to a specialised agent; this skill decides *which* agent, *when*, and *what to hand it*. + +Invoked with an id (`/backlog-task F-02`) it runs that task. Invoked bare (`/backlog-task`) it reads the dispatch order and current sprint and *proposes* the next task, then stops for confirmation — it never picks and starts in one move. + +## When to use + +- A backlog task id is named, or the user asks what to pick up next. +- A bug report arrives that the backlog already owns (check before opening an editor). +- Someone is about to "just fix" something the backlog cites — the premise has not been verified yet. + +**Not** for: authoring an ADR alone (use `write-adr`), a pure research question (`fuse-engine-researcher` directly), or reviewing a change already built (`fuse-code-reviewer` directly). + +## Ground truth + +Read these; do not work from memory or from a summary of them. + +| File | Authoritative for | +| --- | --- | +| `BACKLOG_V2.md` (repo root) | Task ids, tiers, sizes, dependencies, "Dispatch order" (§ near the end), sprints, the HARD GATE, and "Rules for the implementing agent" (final section) | +| `.agents/agents/*.md` | The pipeline agents' own contracts — input, output shape, hard boundaries | +| `.agents/rules/*.mdc` | 13 numbered coding rules (`01-`…`13-`) plus `README.mdc`; all 14 carry `alwaysApply: true` | +| `docs/adr/` | 33 ADRs (`0001`–`0033`), MADR 3.0, index in `docs/adr/README.md` | +| `.specify/memory/constitution.md` | Governing principles; I Test-First and II Quality Gates are NON-NEGOTIABLE | +| `AGENTS.md` | Learned workspace facts + the stage→agent map. Prose can drift from the tree; see Traps | + +**Backlog naming.** The canonical queue is the repo-root file titled `# FUSE — product shape and backlog`, currently `BACKLOG_V2.md`. If the maintainer has renamed it to `BACKLOG.md`, that title identifies the canonical one. The older superseded backlog renumbered **every** id — never carry an id across files. + +**Authoring location.** Guidance is authored under `.agents/` only. `.claude/{rules,skills,commands,agents}` and `.cursor/{rules,skills,commands}` are symlinks into it (ADR-0009) — writing through them corrupts the source of truth. + +## The pipeline + +### 0 · Locate and scope + +1. Read the task's full entry from the repo-root backlog. Quote its body and "Done when" verbatim into your working notes — every later stage works from your restatement. +2. Restate **tier · size · dependencies** from the entry (sizes: S ≈ under a day, M ≈ a few days, L ≈ a week with design risk). +3. Check the "Dispatch order" graph and the sprint lists. Current sprints, verbatim from the backlog: + - **Sprint 1 — prove the engine:** F-01, F-02, F-03, A-07, A-06. + - **Sprint 2 — write the spec:** B-01, plus A-01, A-04, A-05 in parallel. + - **Sprint 3 — the pillar:** B-02, B-05, then B-06. +4. **Refuse to start a task whose blockers are open**, and name them. Dependencies as declared in the backlog entries themselves (re-read the entry; do not trust this list alone): **F-02 blocks all of Tier A**; A-02→F-02; A-03→A-02, F-02, B-01; A-06→F-01.3; B-02→B-01, A-04, A-10; B-03→B-01; B-04→B-03; B-05→B-01; B-06/B-07→B-02, B-05; B-08→B-01; B-10→A-01; C-01→B-03; C-04→A-08, C-03; C-05→A-11; C-10→B-01; C-12→F-02; all of D-* → Tier A green. Non-dependency ordering notes the entries also carry: A-07 "do first"; A-11 "ADR before code"; C-02 "overlaps A-11(a) — do them together". +5. **HARD GATE:** *do not choose an agent execution model before F-02 is green.* If the task or the conversation is drifting toward that decision and F-02 has not landed, stop and say so. + +Bare invocation stops here with a proposal: id, why it is next (dispatch order + sprint), its blockers' status, and its size. + +### 1 · Verify the premise — the stage that can end the pipeline + +Every task cites a *report about* the code, not the code. Delegate to **`fuse-premise-verifier`**. It writes and **executes** a failing test and returns `REPRODUCED` / `NOT-REPRODUCED` / `PARTIAL` / `BLOCKED` with verbatim output. + +- `REPRODUCED` → continue to stage 2 or 3; that test is the first commit of the PR. +- `PARTIAL` → re-scope the task to the clause that reproduced and record which clause did not. +- **`NOT-REPRODUCED` → the pipeline ENDS here.** Close the task in the backlog with the verifier's evidence (file:line showing why the code does not do what the entry says). This is a legitimate, expected, valuable outcome — the backlog's own header says so. +- `BLOCKED` → supply what is missing (a DSN, Docker, a kill point) and re-run; do not proceed on an unverified premise. + +Several premises were **read as diverging** from the code (see Traps) — read, not executed. Expect closures and re-scopes. + +### 2 · Research + +Only when the question spans subsystems, or when the plan would otherwise guess. Delegate to **`fuse-engine-researcher`** — read-only, returns hop-by-hop control flow with every claim marked `CONFIRMED-BY-READING` or `INFERRED`. Load the skill pack named in the routing table *before* reading code, so the patterns are in context. + +Skip it for a single-file change whose premise the verifier already proved. + +### 3 · Plan + +Delegate to **`fuse-implementation-planner`**. It refuses to plan an unverified premise, so stage 1 must have run. Its plan is shown to the maintainer **before any production code is written**. + +Decide the decision-record weight here: + +| Situation | Instrument | +| --- | --- | +| Costly to reverse, shapes the architecture (A-03 delivery guarantee, A-11 output addressing, A-05 version pinning, B-03 registry) | **ADR** via the `write-adr` skill — MADR 3.0 from `docs/adr/template.md`, index row in `docs/adr/README.md` | +| New subsystem or public contract needing requirements + tasks (a new SDK surface, a whole capability registry) | **SpecKit** — `.agents/commands/speckit.*`, creates a `NNN-short-name` branch and `specs/NNN-*/` | +| An ordinary bug fix, a config guard, a plumbing task | **Neither.** A `feat/` or `fix/` branch and a PR body | + +Full SpecKit ceremony is overkill for a backlog task; the two existing specs (`specs/001-ai-agent-node`, `specs/002-anthropic-provider`) are both feature-sized. Note SpecKit's `NNN-short-name` branch naming contradicts the `feat/`/`fix/` prefixes in `.agents/rules/11-development-workflow.mdc`: `.specify/scripts/bash/common.sh:check_feature_branch` errors out ("Not on a feature branch") on any branch that does not match `^[0-9]{3}-`. + +**Protocol tasks invert the order.** If the task touches the remote-node protocol surface, the **spec comes first** — route to `fuse-protocol-spec-author` and land B-01 before any Go. B-01 explicitly "blocks every other B task · do this before writing any Go". + +### 4 · Implement + +Delegate to **`fuse-go-implementer`**, or **`fuse-e2e-harness-engineer`** for F-02-class work (real process, SIGKILL, restart, CI stage). The reproducing test lands **first, failing, in the first commit** — this is a constitution-level rule (Principle I), not a preference. + +### 5 · Review + +Delegate to **`fuse-code-reviewer`** (read-only; it reports, it never fixes). Loop back to stage 4 until every finding is resolved, or explicitly deferred with a written reason recorded in the PR body. A `BLOCKER` finding is never deferred. + +### 6 · Gates and land + +1. `make lint && make build && make test` — in that order, no exceptions (`.agents/rules/12-quality-gates.mdc`). `./scripts/pre-commit-gates.sh` runs the same three quietly. +2. Add whatever the gates do not cover: `go test -race ./internal/... ./pkg/...` for actor/concurrency work, `make test-functional` with `DB_POSTGRES_DSN` exported for anything persistence-shaped, `make e2e-local` for engine-boundary work. +3. Branch prefix `feat/ fix/ docs/ refactor/ test/ chore/`; Conventional Commits subjects (the release version is derived from them). +4. **One task, one PR.** +5. Public API change → changelog entry + migration path. **There is no `CHANGELOG` file in this repo today** — raise that with the maintainer rather than inventing one. +6. HTTP/API change → Swagger annotations + `make swagger`, then `docs/API.md`, then the README route summary only if the public surface changed (`docs/CONTRIBUTE.md`). +7. **Update the backlog**: mark the task done (or closed with evidence), and **append** every drive-by bug found — with file:line evidence — instead of fixing it. + +## Agent invocation + +| Agent | Invoke at | Pass it | Returns | Writes files? | +| --- | --- | --- | --- | --- | +| `fuse-premise-verifier` | Stage 1, always | The task id + the premise quoted verbatim, split into numbered sub-claims | `VERDICT`, `EVIDENCE` (test path, exact command, verbatim output, driver), `CODE FACTS`, `DIVERGENCE FROM BACKLOG`, `SUGGESTED NEXT ACTION`, `NEW BUGS FOUND` | Yes — test files only | +| `fuse-engine-researcher` | Stage 2, when the question spans subsystems | One question, decomposed into sub-questions | `FLOW` hop by hop, `CLAIMS` each marked `CONFIRMED-BY-READING`/`INFERRED`, `NEGATIVE RESULTS`, `DRIVER PARITY`, `DOC / ADR DRIFT`, `OPEN QUESTIONS` | **No** | +| `fuse-implementation-planner` | Stage 3, after a `REPRODUCED`/`PARTIAL` verdict — **but never first for B-01 or any wire-surface change; those start at `fuse-protocol-spec-author` (stage 3, "Protocol tasks invert the order")** | The task id **and** the verifier's full verdict block | `FILES`, `TESTS` (failing-first named), `PARITY`, `HA`, `MIGRATION`/`PUBLIC API`/`CHANGELOG`, `DECISION RECORD`, `PROTOCOL IMPACT`, `OUT OF SCOPE`, `DONE WHEN`, `GATE COMMANDS` | **No** | +| `fuse-go-implementer` | Stage 4 | The plan, verbatim | `STATUS`, `RED TEST` (test-only commit sha + verbatim failure), `CHANGES`, `GATE` (lint/build/test/test-functional/race, each PASS·FAIL·SKIPPED), `GREEN TEST`, `PARITY`, `PROTOCOL`, `OUT OF SCOPE HELD`, `BACKLOG ADDITIONS` | Yes | +| `fuse-e2e-harness-engineer` | Stage 4 for F-02-class work | The plan plus the scenario list from F-02 | `STATUS`, `RIG` (binary, DSN, object store, port), `FILES`, `SCENARIOS` (per scenario GREEN·RED·BLOCKED + owning task id + probe + assertion), `CI`, `ENGINE BUGS FOUND`, `NOT COVERED` | Yes | +| `fuse-code-reviewer` | Stage 5 | The diff scope (working tree / branch vs `main` / PR number) + the task id | `VERDICT`, `GATES`, `NOT COVERED BY GATES`, ranked `FINDINGS` each with a concrete scenario, `TEST-FIRST EVIDENCE`, `SCOPE`, `PROTOCOL CONTRACT`, `OBLIGATIONS`, `BACKLOG ADDITIONS` | **No** | +| `fuse-protocol-spec-author` | Before any Tier B Go | The protocol surface being touched | `STATUS`, `SPEC` (path + version), `DECISIONS` (all eight B-01 questions, none `TBD`), `ADR`, `ENGINE FACTS CITED`, `ENGINE GAPS THIS SPEC CREATES`, `UNBLOCKS`/`BLOCKS-UNTIL-PUBLISHED` | Yes — **Markdown only** (spec doc, ADR, ADR index row); never engine Go | + +All seven were present under `.agents/agents/` when this file was written. Still run `ls .agents/agents/` before delegating — the roster is authored, not shipped, and a missing agent means doing that stage inline, not skipping it. The `Returns` column above is transcribed from each agent's own `## Output format` block; if an agent file is revised, re-sync this table from it rather than trusting this copy. + +## Routing table — every task in the backlog + +`Premise` column: `VERIFY` = report-derived, run stage 1 (the default for every task). `VERIFY ⚠` = the entry's wording diverges from what the code was read to do — see Traps. `CONFIRMED` = the cited code fact was read directly in this repo, but the *fix* still needs the failing test first. `net-new` = nothing to reproduce; it is new construction. + +`Skill pack` column: prefer the **FUSE-specific** packs — `durable-execution-internals`, `crash-resume-testing`, `persistence-and-migrations`, `observability-tracing`, `remote-node-protocol`, `capability-registry-and-agents`, `function-package-authoring`. Each carries `name`/`description` frontmatter and names the backlog ids it owns; they describe *this* engine. The rest (`actor-model-patterns`, `ddd-patterns`, `clean-hexagonal-architecture`, `microservices-architecture`, `cqrs-event-driven`, `go-concurrency`, `go-senior-developer`) are generic, frontmatter-less reference material — reach for them only as a supplement. + +### Tier F — foundation + +| id | intent | primary code area | skill pack | premise | +| --- | --- | --- | --- | --- | +| F-01 | Verify three findings: discarded span ctx; duplicate subworkflow child on restart; shared graph pointer | `internal/actors/workflow_func.go:96`; `internal/workflow/workflow.go:findPendingThreads/replayPendingThread`; `internal/repositories/graph_memory.go:FindByID` | observability-tracing (.1), durable-execution-internals (.2), persistence-and-migrations (.3) | this task *is* stage 1 | +| F-02 | Crash-resume e2e harness: real process, real Postgres, fs store, SIGKILL, restart, assert | `tests/e2e/`, `docker-compose.yml` (`e2e` profile), `.github/workflows/e2e.yml` (jobs `e2e-fast`, `e2e-slow`); assertion target `internal/actors/workflow_handler.go:156` | crash-resume-testing + durable-execution-internals + `.agents/rules/07-testing.mdc` | CONFIRMED — `Resume()` has exactly one call site (`workflow_handler.go:156`) and no test caller | +| F-03 | Reject persistent DB + ephemeral object store in config validation | `internal/app/config/config.go:Validate`; `internal/app/di/repos.go`; `internal/app/di/objectstore.go` | persistence-and-migrations | CONFIRMED — `Validate` checks only the `CLUSTER_ETCD_ENDPOINTS` pairing and returns `nil` otherwise | + +### Tier A — core correctness + +| id | intent | primary code area | skill pack | premise | +| --- | --- | --- | --- | --- | +| A-01 | Input payload on trigger, exposed to `SourceFlow` under `trigger.*` | `internal/dtos/workflow.go`; `internal/messaging/trigger_workflow.go`; `internal/actors/workflow_sup.go:spawnWorkflowActor`; `internal/workflow/workflow.go:Trigger` | function-package-authoring + ddd-patterns | VERIFY | +| A-02 | Replay the intercepted system functions (subworkflow, awakeable, sleep) | `internal/workflow/workflow.go:replayJournalEntries`; `internal/actors/workflow_handler.go:handleSystemSleep`/`handleSystemWait`/`handleSystemSubWorkflow` | durable-execution-internals + actor-model-patterns | VERIFY ⚠ awakeable clause | +| A-03 | Decide + enforce replay semantics for pending remote steps (ADR) | `internal/workflow/workflow.go:findPendingThreads/replayPendingThread`; `internal/handlers/async_function_result.go` | durable-execution-internals + remote-node-protocol + write-adr | VERIFY ⚠ execID clause | +| A-04 | Trace context end to end; add `context.Context` to `ExecutionInfo` | `internal/actors/workflow_func.go:96`; `internal/tracing/provider.go:ExtractCarrier`; `pkg/workflow/execution_info.go`; `internal/handlers/trigger_workflow.go` | observability-tracing | CONFIRMED — `_ = nodeCtx` at `workflow_func.go:96` | +| A-05 | Pin runs to a schema version; leave in-flight runs alone on rollback | `internal/repositories/postgres/workflow.go:loadGraph`; `internal/repositories/postgres/migrations/`; `internal/workflow/versioned_schema.go`; `internal/services/graph_service.go` | persistence-and-migrations + write-adr | VERIFY | +| A-06 | Deep-copy from the memory graph repo; assert driver parity | `internal/repositories/graph_memory.go:FindByID`; `tests/functional/graph_repository_test.go:contractTestGraphRepository` | persistence-and-migrations | VERIFY (= F-01.3) | +| A-07 | Suppress journal append during replay via a replay flag | `internal/workflow/workflow.go:SetResultFor`; `internal/workflow/journal.go:LoadFrom`/`NewEntries` | durable-execution-internals | VERIFY ⚠ trace clause | +| A-08 | Attempt counter on the journal entry; rebuild `RetryTracker` on replay | `internal/workflow/retry_tracker.go`; `internal/workflow/workflow.go:HandleNodeFailure`; `internal/workflow/trace_builder.go` | durable-execution-internals | VERIFY | +| A-09 | Inline small payloads; content-address large ones | `internal/repositories/postgres/journal.go:Append/LoadAll`; `internal/repositories/postgres/workflow.go:Save`; `internal/repositories/postgres/trace.go`; `pkg/objectstore/` | durable-execution-internals + persistence-and-migrations | VERIFY | +| A-10 | Configurable pool size; remote calls must not hold a worker | `internal/actors/workflow_func_pool.go:40`; `internal/actors/workflow_instance_sup.go` | go-concurrency + actor-model-patterns (+ remote-node-protocol for the non-blocking half) | CONFIRMED — `PoolSize: 3` hardcoded in `WorkflowFuncPool.Init` | +| A-11 | Address node output by execution — **ADR before code** | `internal/workflow/workflow.go:428` (`w.aggregatedOutput.Set(entry.FunctionNodeID, …)`); `pkg/store/kv.go` | capability-registry-and-agents + durable-execution-internals + write-adr | CONFIRMED — keyed by node id, not exec id | + +### Tier B — extension protocol and SDKs + +**Nothing in Tier B is shipped behaviour.** The backlog states it plainly: "everything here is net-new". The engine today declares `HTTP` and `gRPC` in `pkg/transport/type.go` and implements neither; `PackagedFunction.Function` is `json:"-"`, so an API-registered function is metadata-only and `LoadedPackage.ExecuteFunction` returns `function %s has no transport`. The `primary code area` column below names *where the proposed work would land, or the existing raw material it builds on* — never a capability you can use now. Do not describe any of it to a user as something FUSE does. + +| id | intent (**all proposed**) | primary code area / raw material | skill pack | premise | +| --- | --- | --- | --- | --- | +| B-01 | **Spec document** for the remote node protocol — blocks every other B task | proposed new doc under `docs/`; raw material in `pkg/transport/type.go`, `internal/packages/transport/function.go`, `pkg/workflow/execution_info.go`, `internal/handlers/async_function_result.go`, `pkg/workflow/fn_result.go:NewFunctionResultAsync` | remote-node-protocol + write-adr | net-new (spec, not a defect) | +| B-02 | Implement the HTTP transport; validate the callback's exec is pending | `internal/packages/loaded_package.go:41` and `:MapToRegistryPackage`; `internal/packages/transport/`; `internal/packages/loaded_function.go` | remote-node-protocol + function-package-authoring | CONFIRMED (of the *gap*, not the feature) — `loaded_package.go:41` is the "has no transport" dead end; `async_function_result.go:HandlePost` does no pending-state or duplicate validation | +| B-03 | Capability registry: one registration → node **and** tool projection | `internal/packages/registry.go`; `internal/packages/agent_tools.go` (today an adapter over the same `Registry`, not a second projection); `internal/services/package_service.go` | capability-registry-and-agents + remote-node-protocol | net-new construction; VERIFY the projection-parity premise against `agent_tools.go` first | +| B-04 | MCP ingestion — import an MCP server's tools as capabilities | proposed new package under `internal/packages/`; `internal/dtos/package.go` | capability-registry-and-agents | net-new | +| B-05 | SDK conformance suite, built before the first SDK | proposed new test tier; model it on `tests/e2e/` + `.github/workflows/e2e.yml` | remote-node-protocol + crash-resume-testing | net-new | +| B-06 | SDK: TypeScript / Node (plain + NestJS) | **outside this repo** — built against the B-01 spec and the B-05 suite | remote-node-protocol (spec side only) | net-new | +| B-07 | SDK: PHP / Laravel | **outside this repo** — built against the B-01 spec and the B-05 suite | remote-node-protocol (spec side only) | net-new | +| B-08 | Worker registration lifecycle: register, health, deregister, drain | `internal/services/package_service.go`; `internal/actors/mux_worker.go` (the HTTP route table — where lifecycle routes would be declared); `internal/repositories/postgres/package.go` | remote-node-protocol | net-new; VERIFY what registration exists today before designing around it | +| B-09 | Guard sub-workflow recursion: depth counter, max, cycle detection | `internal/actors/workflow_handler.go:handleSubWorkflowAction`; `internal/workflow/subworkflow.go` | durable-execution-internals (the `subworkflow:*` entries and the spawn path) + actor-model-patterns | VERIFY (the absence of a guard) | +| B-10 | Pass input into sub-workflows; add an output selector | `internal/actors/workflow_handler.go:handleSubWorkflowAction`; `internal/messaging/trigger_workflow.go`; `internal/workflow/workflow.go:AggregatedOutputSnapshot` | actor-model-patterns + durable-execution-internals | VERIFY | + +### Tier C — agents in Core + +**Tier C is proposed work too.** `ai/agent` and `ai/chat` exist as async node functions, but every capability named below — registry binding, first-class memory, a model gateway, GenAI cost on the trace, a typed human-gate payload, durable timers, an awakeable-id route, `GET /v1/workflows/{id}/state` — is a task, not a feature. Where the premise column says `CONFIRMED`, what is confirmed is the **gap** (a symbol that exists with no caller, a route that is absent), never the proposed replacement. + +| id | intent (**all proposed**) | primary code area / raw material | skill pack | premise | +| --- | --- | --- | --- | --- | +| C-01 | Bind the agent node to registry capabilities; identical journal + trace | `internal/packages/functions/ai/agent.go`, `tools.go`; `internal/packages/agent_tools.go` | capability-registry-and-agents | VERIFY | +| C-02 | Memory as a first-class concept (working / episodic / semantic) | `internal/packages/functions/ai/context.go`; proposed new repository | capability-registry-and-agents + write-adr | net-new; backlog says "overlaps A-11(a) — do them together" | +| C-03 | Model gateway: provider routing, budgets, redaction | `internal/llm/providers/` (`anthropic`, `openaicompat`) | capability-registry-and-agents + microservices-architecture | net-new | +| C-04 | GenAI telemetry and cost on the trace | `internal/packages/functions/ai/usage.go`; `internal/tracing/provider.go`; `internal/metrics/` | observability-tracing | VERIFY · declared deps A-08 + C-03 | +| C-05 | Human gate payload schema + correction semantics | `internal/workflow/awakeable.go`; `internal/packages/functions/system/wait.go`; `internal/handlers/resolve_awakeable.go` | capability-registry-and-agents + durable-execution-internals | net-new; the entry declares it depends on A-11 being resolved | +| C-06 | Durable timers and an expiry sweeper; re-arm on boot, claim-aware | `internal/actors/execution_timer.go`; `internal/actors/workflow_handler.go`; `internal/actors/workflow_claim_actor.go`; `Awakeable.DeadlineAt` (`internal/workflow/awakeable.go:31`) | durable-execution-internals + actor-model-patterns + go-concurrency | VERIFY | +| C-07 | Expose awakeable IDs over the API | `internal/repositories/awakeable.go:16` (`FindPending`); `internal/actors/mux_worker.go`; `internal/handlers/get_workflow.go` | durable-execution-internals + clean-hexagonal-architecture | CONFIRMED — `FindPending` is declared on the interface and implemented by both drivers; no route in `mux_worker.go` exposes it | +| C-08 | ForEach state survives restart; stop the silent interception bypass | `internal/workflow/foreach.go`, `foreach_state.go`; `internal/actors/workflow_handler.go:spawnForEachBatch` (line 1013) | durable-execution-internals + persistence-and-migrations + actor-model-patterns | CONFIRMED — plus a blocking enum gap, see Traps | +| C-09 | Mid-flight run read model (proposed `GET /v1/workflows/{id}/state` — no such route today) | `internal/handlers/get_workflow.go`, `get_workflow_snapshot.go`; `internal/workflow/execution_snapshot_builder.go`; `internal/events/memory_bus.go` | durable-execution-internals + cqrs-event-driven | VERIFY | +| C-10 | Per-node idempotency key from `(runID, execID, attempt)` | `internal/idempotency/store.go`; `pkg/workflow/execution_info.go` | remote-node-protocol | VERIFY · declared dep is B-01; that the `attempt` component must come from A-08 is inference, not a backlog-declared dependency | +| C-11 | Claim failure must fail closed | `internal/actors/workflow_handler.go:369` (`claimForThisNode`); `internal/repositories/postgres/claim.go`; `internal/actors/workflow_claim_actor.go` | actor-model-patterns + persistence-and-migrations | CONFIRMED — on a claim-store error it logs "running anyway" and returns `true` | +| C-12 | Multi-thread resume correctness | `internal/workflow/workflow.go:226` (`buildResumeAction`) | durable-execution-internals + crash-resume-testing | VERIFY ⚠ backlog itself flags it "premise is inference" | + +### Tier D — Enterprise layer + +`D-01` tenancy/RBAC/SSO · `D-02` console · `D-03` collaboration · `D-04` cost dashboards · `D-05` audit export · `D-06` packaging. + +**All D-* are Enterprise repo — not implemented in `core/`.** They start once Tier A is green and consume Core's public API only. If a task in this repo only makes sense with the commercial layer present, it belongs in the commercial layer, not here. + +## Non-negotiable pipeline invariants + +Restated from the backlog's "Rules for the implementing agent". These are not guidance; a stage that violates one is wrong. + +1. **Verify before fixing.** Every task cites a report, not the code. If the premise does not hold, close the task and report it — that is a success. +2. **One task, one PR.** They are deliberately separable. +3. **A test reproducing the bug lands before the fix**, in the same PR, failing in the first commit. +4. **Do not fix bugs found along the way.** Append them to the repo-root backlog with evidence and keep going. +5. **The protocol spec is a contract.** Once B-01 is published and an SDK exists, changing it breaks strangers. Treat it with more care than engine internals — spec first, code second, versioned. +6. **Core stays usable without Enterprise.** +7. **Public API changes need a changelog entry and a migration path.** Apache-2.0, with users who are not you. + +Plus, from this repo: gates run **lint → build → test** in that order; author under `.agents/` only; never widen the quarantined `"untriggered"` retry in `tests/e2e/workflow_fixture_e2e_test.go` to make a suite green. + +## Traps — what looks right and is wrong + +Each **code fact** below was read in this repo at the cited symbol. The **runtime consequences** drawn from those facts ("each restart adds a row", "two callbacks race") are reasoning over the control flow, not observations — nothing here has been executed under a real SIGKILL, because F-02 is what would make that possible. Treat the code facts as load-bearing and the consequences as the hypothesis stage 1 must confirm. They are the reason stage 1 exists. + +- **A-03's execID clause is contradicted by the code.** The backlog says the worker is re-invoked "with a new execID, and the original callback is orphaned". Read at `internal/workflow/workflow.go:295`: `replayPendingThread` sets `FunctionExecID: workflow.ExecID(pt.execID)` — the **original** execID carried out of the journal by `findPendingThreads`, not a new one — so the pre-crash callback URL still routes. *Inferred:* the actual exposure is a duplicate dispatch with two callbacks able to arrive for one execID, since `internal/handlers/async_function_result.go:HandlePost` (read in full) does no pending-state, duplicate or staleness validation — it binds the path params and sends the message. Re-scope before implementing. +- **A-02's awakeable clause looks wrong as written.** The backlog says the token you handed out "is orphaned pending forever". Read: `internal/handlers/resolve_awakeable.go:66` resolves via `awakeableRepo.FindByID(awakeableID)` and then sends the awakeable's *own stored* `ExecID`/`ThreadID`, so the pre-crash token still routes after a restart. What the code does show is `handleSystemWait` (`workflow_handler.go:727`) minting `awakeableID := uuid.New().String()` on **every invocation**, with no read of `JournalAwakeableCreated` anywhere. *Inferred from that:* since resume re-issues pending steps and `system/wait` is intercepted, each restart should add an extra awakeable row plus an extra `awakeable:created` entry and re-arm the full original timeout. Verify both halves — the failing clause and the replacement — before re-scoping. +- **A-07's trace clause is contradicted by the code.** The backlog says `BuildTrace` "renders each step N+1 times after N restarts". Read at `internal/workflow/trace_builder.go:10`: a step row is appended **only** on `JournalStepStarted`, and `replayJournalEntries` (`workflow.go:199`) does not append a journal entry for that type — it only calls `auditLog.NewEntry`. It is `JournalStepCompleted` that routes into `SetResultFor`, which *does* append. So a duplicated `step:completed` finds the existing `stepIdx[execID]` and mutates that row in place: `CompletedAt`/`Duration` recomputed against the **replay** timestamp and `Status` forced back to `completed`. Corruption in place, not duplication. The journal-growth half of the premise stands. Critical constraint on the fix: line 428's `w.aggregatedOutput.Set(...)` inside `SetResultFor` is the only thing rebuilding aggregated output during replay — gate the append, never skip the call. +- **C-12 is narrower than written.** Read at `internal/workflow/workflow.go:226`: `buildResumeAction` returns the first non-noop action **only on the zero-pending branch**; with one pending thread it replays that thread, and with several it fans out correctly into a `RunParallelFunctionsAction`. So the reproducing scenario is specifically a crash at a fan-out/join boundary with nothing still in flight — not "any multi-thread crash". `Next()` also mutates state and appends `thread:finished`, so actions discarded by that loop have already had side effects; *inferred from that*, the bug may present intermittently. Do not encode that guess in the test — let the F-02 multi-thread scenario settle it. +- **C-08 has a blocking prerequisite the entry does not mention.** Confirmed by reading both sides: `internal/workflow/journal.go` declares 18 `JournalEntryType` constants including the four `foreach:*` values (lines 43–49), while `migrations/000001_create_tables.up.sql:12` creates `journal_entry_type` with **13** values and no `foreach:*` among them; the only later widening is `000004_add_manual_retry_journal_type.up.sql`, which adds `step:manual-retry` alone. *Inferred (not executed):* with `DB_DRIVER=postgres`, inserting a `foreach:*` entry must therefore be rejected by the enum, so a ForEach run's journal flush fails. Ship the `ALTER TYPE … ADD VALUE` migration first — verify this under a real Postgres as the very first step of C-08, since nothing about ForEach replay is observable until it holds. +- **A green `make test` proves less than it looks.** No `-race` in the Makefile or in `.github/workflows/ci.yml`. `make test-functional` **skips silently** without `DB_POSTGRES_DSN` (`tests/functional/postgres_test.go:29` calls `t.Skip`). E2E is a separate `workflow_run` workflow gated on CI success, and its `e2e-slow` job additionally requires `head_branch == 'main'` — so the slow tier only ever runs after merge. `scripts/check-coverage.sh` is referenced by neither the Makefile nor any workflow. `.git/hooks/` holds no installed hook in this working copy. +- **A fresh clone cannot build.** `docs/docs.go` and `docs/swagger.{json,yaml}` are gitignored but blank-imported by `internal/actors/mux_server.go`. Run `make swagger` first, or `make build` *and* `make test` both fail. +- **Never validate a resume fix against the memory driver.** `MemoryWorkflowRepository.Get` returns the stored `*Workflow` pointer as-is, with whatever threads/audit log/aggregated output the live object accumulated; `postgres/workflow.go:Get` calls `workflow.New(...)` and restores only `state`, so it hands back a fresh object. A resume test that passes under memory proves nothing about Postgres. Postgres also carries `CREATE UNIQUE INDEX idx_journal_wf_seq ON journal_entries (workflow_id, sequence)` (migration `000001`); memory has no equivalent. +- **Trust `mux_worker.go` and the `Makefile` over prose.** Two facts that older docs got wrong, both re-checked here: example seeding is `./bin/fuse seed examples --ci` (flag declared at `internal/app/cli/seed.go:56`) — there is no `examples-ci` make target and no `scripts/run-example-workflows.sh`. And `GET /v1/workflows/{workflowID}/status` **is not a route**: `internal/actors/mux_worker.go:220` registers `/v1/workflows/{workflowID}`, whose handler returns `dtos.GetWorkflowResponse` — `{workflowId, status}` and nothing else. Richer mid-flight state is C-09. +- **`.golangci.yml` declares `version: 2` but carries v1 schema keys.** Confirmed by reading: the file opens with `version: 2` and still uses `linters-settings:` (with `gocyclo.min-complexity: 15` and `govet.check-shadowing`), `run.skip-dirs`, `issues.exclude-rules` and `output.format` — all of which v2 renamed or moved. **Inferred, not observed** (golangci-lint was not executed to check): v2 therefore does not apply them, so the documented complexity ceiling and the "no errcheck/gosec on tests" exclusion are probably not in force. Write tests defensively (`defer func() { _ = f.Close() }()`), and do not casually "fix" the config mid-task — repairing it likely surfaces a wave of findings across the tree, which is a separate task. + +## Exact commands + +```bash +make swagger # FIRST on a fresh clone; generates the gitignored docs package +make lint && make build && make test # the mandated gate, in this order +./scripts/pre-commit-gates.sh # same three, quiet +go test -v -run TestName ./internal/workflow/ # single test +go test -race ./internal/... ./pkg/... # NOT in any gate — run it for actor/concurrency changes +go clean -testcache # replay tests are order/timing sensitive + +make infra-up # docker compose --profile infra (PG + S3 + etcd) +DB_POSTGRES_DSN='postgres://fuse:fuse@localhost:5432/fuse?sslmode=disable' make test-functional +make migrate # build, then ./bin/fuse migrate +make e2e-local # build image, compose --profile e2e, tags=e2e against :9091, tear down +make ha-up / make ha-down # 3-node HA cluster from source +make test-benchmark + +ls docs/adr/[0-9][0-9][0-9][0-9]-*.md | sed -E 's#.*/([0-9]{4})-.*#\1#' | sort -n | tail -1 # next ADR number +ls .agents/agents/ # which pipeline agents are installed +``` + +Docker profiles are `infra`, `ha`, `e2e` in the single `docker-compose.yml`. + +$ARGUMENTS diff --git a/.agents/skills/capability-registry-and-agents/SKILL.md b/.agents/skills/capability-registry-and-agents/SKILL.md new file mode 100644 index 0000000..f8aef8b --- /dev/null +++ b/.agents/skills/capability-registry-and-agents/SKILL.md @@ -0,0 +1,277 @@ +--- +name: capability-registry-and-agents +description: >- + Maps what FUSE's agent machinery actually is today — ai/agent and ai/chat's real parameters, the provider + registry, token accounting, structured output, the tool-catalog adapter — and the three structural gaps the + capability registry must close: node/tool projection parity, per-execution addressing of node output, and + durable human gates. Use when working B-03, B-04 or any Tier C agent task, or before assuming an agent tool + call is traced and journalled like a node call. +--- + +# capability-registry-and-agents + +Agents now run **inside the engine**. That is what promotes A-09, A-10 and A-11 into Tier A, and it +is why the capability registry (B-03) has to be designed before either projection is built. + +Backlog ids are from the repo-root backlog (`BACKLOG_V2.md`). *If it has been renamed to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog."* + +> **§ TODAY** = shipped behaviour, `file:line` traceable. **§ PROPOSED** = backlog design that does +> not exist. Never quote a § PROPOSED option as engine behaviour. + +## When to use + +- **B-03** capability registry, **B-04** MCP ingestion. +- **C-01** agent↔registry tool binding, **C-02** memory, **C-04** GenAI telemetry, **C-05** human + gate payloads, **C-06/C-07** durable timers and awakeable IDs. +- **A-11** per-execution addressing of node output — the single most expensive item in the backlog + to reverse. +- The wire half of a registered capability lives in `.agents/skills/remote-node-protocol/SKILL.md`. + +## Ground truth + +| File | Authoritative for | +| --- | --- | +| `internal/packages/functions/ai/agent.go` | The whole reasoning loop and `ai/agent`'s declared parameters | +| `internal/packages/functions/ai/chat.go` | `ai/chat`, provider resolution, temperature parsing | +| `internal/packages/functions/ai/tools.go` | The `ToolRegistry` port + function-metadata→JSON-Schema conversion | +| `internal/packages/agent_tools.go` | The adapter that turns the package registry into the tool catalog | +| `pkg/llm/provider.go` / `registry.go` | The entire public LLM contract and the per-environment registry | +| `internal/app/di/llm.go` | The only place providers are named and enabled | +| `internal/metrics/registry.go` | Where LLM usage lands, and its label set | +| `internal/workflow/workflow.go` | `aggregatedOutput` producer/consumer — the A-11 crux | +| `pkg/store/kv.go` | The objx dot-path KV that *is* `aggregatedOutput` | +| `internal/workflow/awakeable.go` + `internal/handlers/resolve_awakeable.go` | The human-gate primitives | +| `docs/adr/0007`, `0026`, `0027`, `0028`, `0029`, `0030` | Tools-from-functions, agent-as-orchestrator (Proposed), async tool invocation (Proposed), context/memory, cost, structured output | + +--- + +## § TODAY — the parity question, answered + +**B-03's hard design requirement:** an agent invoking a capability and a graph node invoking the same +capability must produce the same journal entries, the same span shape and the same idempotency +behaviour. + +**That parity is FALSE today — by construction, not by omission.** This is the single most +decision-relevant fact in this pack. + +| | Graph node path | Agent tool path | +| --- | --- | --- | +| Entry point | `internal/actors/workflow_func.go:63 HandleMessage` | `ai/agent.go:271 e.tools.InvokeTool(...)` | +| Adapter | — | `internal/packages/agent_tools.go:79 InvokeTool` (linear scan of `registry.List()`) | +| Execution | `LoadedPackage.ExecuteFunction` → `InternalFunctionTransport.Execute` | `LoadedPackage.ExecuteFunctionSync` (`loaded_package.go:50`) → `ExecuteSync` (`internal.go:68`) | +| Span | `node.execute` created at `workflow_func.go:90` | **none** | +| Journal | `step:started` at dispatch (`newRunFunctionAction`, `workflow.go:616`; also `:171` on trigger, `:488` on manual retry) + `step:completed`/`step:failed` via `SetResultFor` (`workflow.go:422`) | **none** | +| Metric | `NodeExecDuration{function_id,status}` via `recordNodeDuration` (`workflow_func.go:181`) | **none** | +| Retry / timeout | node `RetryPolicy` + `ExecutionTimer` | **none** — one inline call, no timeout of its own | +| Idempotency | none (trigger-level only) | none | + +The tool path deliberately never reaches the actor system, so it misses the sole `node.execute` span +site and the sole `step:completed` producer. **The only durable record of a tool call is the `steps` +array inside the agent node's single `FunctionOutput`**, which lands in one `step:completed` journal +entry as an opaque blob (`agent.go:148` — `steps` is a local variable; each entry is built at +`agent.go:282` on success and `:288` on failure). + +Achieving parity therefore requires either journalling from inside the agent goroutine — which today +has no journal handle, because `ExecutionInfo` carries only `WorkflowID`/`ExecID`/`Environment`/ +`Input`/`Finish` — or routing tool calls back through the actor system. ADR-0027 (Proposed) names the +second and constrains how: a **second typed per-execution port** (`ExecRuntime` with +`InvokeAsync(...)`), injected into orchestrating nodes, and explicitly **not** a field on +`workflow.ExecutionInfo` (the earlier `Handle any` field was removed for exactly that reason). + +⚠️ **The tool-eligibility predicate is half-dead.** `isExposableTool` +(`internal/packages/agent_tools.go:95`) requires `fn.Transport != nil && fn.Metadata.Transport == +transport.Internal`. But `MapToRegistryPackage` hardcodes `Transport: transport.Internal` into every +registry-side metadata (`internal/packages/loaded_package.go:70`), so the second clause is vacuously +true and the real discriminator is the nil-transport check. A B-03 predicate written against +`Metadata.Transport` will match everything. + +The catalog also excludes, by construction: the whole `ai` package (`agent_tools.go:53`), any +`CustomParameters` function, and the `interceptedOrAsyncFunctionIDs` denylist (`agent_tools.go:18-24`: +`system/sleep`, `system/wait`, `system/subworkflow`, `system/foreach`, `fuse/pkg/logic/timer`). +An agent can therefore not open a human gate, sleep, spawn a sub-workflow or call another agent — +the exact primitives "agents in Core" needs. A tool that returns `Async` at runtime is rejected +inline with `"tool is asynchronous and not supported by the agent"` (`agent.go:276`). Tool +descriptions are a placeholder: `fmt.Sprintf("FUSE function %s", fullID)` (`agent_tools.go:67`). + +## § TODAY — what `ai/agent` and `ai/chat` actually are + +Both are ordinary internal functions registered by `ai.New(providers, tools, usage)` +(`internal/packages/functions/ai/package.go:18`, `PackageID = "fuse/pkg/ai"`), both **async**: they +return `NewFunctionResultAsync()` and complete via `execInfo.Finish` from a goroutine with its own +`context.WithTimeout(context.Background(), …)` — `agentTimeout = 5 * time.Minute` (`agent.go:26`), +`chatTimeout = 2 * time.Minute` (`chat.go:18`). + +| | `ai/agent` (`agent.go:33`) | `ai/chat` (`chat.go:24`) | +| --- | --- | --- | +| Inputs | `input`(required), `provider`, `model`, `systemPrompt`, `temperature`, `maxIterations`, `allowedTools`, `maxContextTokens`, `contextStrategy`, `outputSchema` | `input`(required), `provider`, `model`, `systemPrompt`, `temperature`, `outputSchema` | +| Outputs | `output`, `usage`, `steps` | `output`, `usage` | +| Loop bounds | `defaultMaxIterations = 10`, `maxMaxIterations = 25` (`agent.go:22-24`) | single completion | + +**Provider abstraction.** `pkg/llm/provider.go:82` — `Provider{Name(); Chat(ctx, ChatRequest)}`. That +is the whole contract: no capability metadata, no pricing. `StreamingProvider` (`:101`) exists with +**zero implementations** — `ChatStream` appears only in the interface declaration. Resolution is +per-`(environment, name)` through `pkg/llm/registry.go`'s `ProviderFactory func(ctx, environment)`, +so ADR-0031 secret refs resolve per execution; `resolveProvider` (`chat.go:178`) falls back to the +registry default. Providers are named at `internal/app/di/llm.go:19-23` and registered only by +`provideLLMRegistry` (`llm.go:45`) — `openai`, `openrouter`, `ollama`, `gemini`, `anthropic`, +**all disabled by default** (`LLMProviderConfig.Enabled`, `config.go:66`, `envDefault:"false"`); +with none enabled it logs `no LLM providers enabled; ai/chat and ai/agent nodes will be unavailable` +(`llm.go:77`). + +⚠️ `LLM_<PROVIDER>_TEMPERATURE` is **dead config**: declared at `internal/app/config/config.go:73` +and read nowhere. The only temperature that reaches a request is the per-node `temperature` input +via `optionalTemperature` (`chat.go:186`). + +**Usage and cost.** `ai.UsageRecorder` (`ai/usage.go:8`) is a two-method port — +`RecordUsage(function, provider, model, llm.Usage)` and `RecordCall(function, provider, model, status)` +— adapted in `internal/packages/usage_recorder.go` onto Prometheus counters +(`internal/metrics/registry.go:67-77`): + +``` +fuse_llm_tokens_total{function, provider, model, type} +fuse_llm_calls_total{function, provider, model, status} +``` + +⚠️ **No workflow id, exec id, node id, tenant or environment label exists on either counter**, and +the recorder writes nowhere else. "What did this run cost" is unanswerable from any FUSE surface +today — that is **C-04**, which is additionally blocked by `workflow_func.go:96` (`_ = nodeCtx` +discards the span context) and by the node span ending at `:166`, before an async node's first LLM +call. + +**Structured output** (ADR-0030, `ai/structured.go`): a synthetic `respond` tool forced with +`ToolChoice: "required"`, up to `structuredRepairAttempts = 2`. Validation is deliberately lenient +(`valueMatchesType`, `:125`): unknown type strings pass, extra fields are never rejected, and +`parseOutputSchema` (`:20`) returns nil on a malformed schema — silently degrading the node to +free-form text rather than erroring. + +**Context bounding** (ADR-0028, `ai/context.go`): `estimateTokens` is `len(chars)/4` +(`approxCharsPerToken = 4`, `:13`) — a heuristic, not a tokenizer. Strategies are `drop-oldest` +(default) and `summarize` (an extra `provider.Chat`). It trims **inside one node execution only**. + +## § TODAY — the state problem (A-11's crux) + +`aggregatedOutput` is a `*store.KV` created at `internal/workflow/workflow.go:65`. + +- **Producer:** `SetResultFor` → `w.aggregatedOutput.Set(entry.FunctionNodeID, result.Output.Data)` + (`workflow.go:428`) — keyed by **node ID**. +- **Consumer:** `applyFlowMapping` → `w.aggregatedOutput.Get(mapping.Variable)` (`workflow.go:751`), + where `mapping.Variable` is `"<nodeID>.<outputParam>"`, split by `strutil.AfterFirstDot` + (`workflow.go:728`). +- **Backing store:** `pkg/store/kv.go` over `objx` — `Set` (`:72`) and `Get` (`:79`) use **dot-path** + semantics; `Snapshot` (`:44`) is `maps.Clone`, a **shallow** copy. + +Three consequences any A-11 design must address, not just the re-keying: + +1. **A node that executes twice overwrites its own prior output.** The second write wins; there is no + history in the data plane, though the journal keeps every execution. +2. **A node id containing a `.` is silently written as a nested key** (objx dot notation), and + `AfterFirstDot` splits the mapping variable on the *first* dot — so a dotted node id also breaks + output-parameter validation. +3. **A retained snapshot shares the inner per-node maps with the live KV**, because `Snapshot` is + shallow. Its consumers are the execution snapshot, sub-workflow completion, and repository `Save`. + +**Why an agent loop makes this fatal:** an agent loop is a cycle; every turn re-executes the same +node; each turn destroys the previous turn's output in the data plane. Conversation history, per-turn +selections, and any "reject item 2, keep 1, 3 and 4" correction are unaddressable from inside the +graph. Compounding it, the transcript itself never leaves the goroutine: `messages`, `totalUsage` and +`steps` are locals in `agentExecutor.run` (`agent.go:146-148`) — confirmed by reading — so nothing +about a turn is durable until the node finishes, and no run can be resumed mid-conversation. + +⚠️ *Inference, not verified — do not build on it.* What a restart **during** an agent turn actually +does is unsettled. An async node is journalled `step:completed` at dispatch (see +`.agents/skills/remote-node-protocol/SKILL.md`, § TODAY), so `findPendingThreads` may not treat the +agent node as pending and may not re-dispatch it at all — which would lose the turn silently rather +than repeat it. Settle it with the **F-02** harness before designing around either outcome. + +### § PROPOSED — A-11's two options (neither exists; an ADR precedes either) + +| Option | Shape | What it costs | +| --- | --- | --- | +| **(a) Turn state as a first-class Core concept** — *the backlog leans here* | A dedicated store addressed by `(runID, agentNodeID, turn)`, entirely separate from `aggregatedOutput` | A new repository beside `internal/repositories/awakeable.go`, new `JournalEntryType` constants **plus** an `ALTER TYPE journal_entry_type ADD VALUE` migration (pattern: `000004_add_manual_retry_journal_type.up.sql`), and a way for the agent goroutine to reach it — which today it cannot. Smaller blast radius, no breaking schema change, and it **overlaps C-02 memory**, so do them together | +| **(b) Re-key `aggregatedOutput` by `(nodeID, execID)`** with a `latest` alias | Correct and general | Breaking-ish; needs a migration story and must also settle the dot-path, `AfterFirstDot` and shallow-snapshot problems above | + +**Do not start by re-keying the KV.** Write the ADR first (`.agents/skills/write-adr/SKILL.md`). +This is the one item in the backlog that is expensive to reverse. + +## § TODAY — human-gate primitives, and what is missing + +| Primitive | Location | State | +| --- | --- | --- | +| `system/wait` metadata | `internal/packages/functions/system/wait.go` | inputs `timeout`, `filter`; outputs `data`, `timedOut`. Body is a placeholder — intercepted by the handler | +| Awakeable row | `internal/workflow/awakeable.go:24` | `{ID, WorkflowID, ExecID, ThreadID, CreatedAt, Timeout, DeadlineAt, Status, Result}` | +| Mint | `internal/actors/workflow_handler.go:728` (`uuid.New()`), journalled at `:785` as `Data{"awakeableId", "timeout"}` | | +| Deadline | `workflow_handler.go:775` — `DeadlineAt: now.Add(action.Timeout)` | ⚠️ **always set**, so a no-timeout gate gets `DeadlineAt == CreatedAt`. A C-06 sweeper treating `deadline_at <= now()` as expired would kill every untimed gate | +| Resolve | `POST /v1/awakeables/{awakeableID}/resolve` — `internal/handlers/resolve_awakeable.go:55` | The **only** awakeable route (`mux_worker.go:270`). It 404s on missing and 400s on non-pending — the validation template the async-exec route lacks | + +⚠️ **The `timedOut` output is never true.** Grep finds exactly three hits: the metadata declaration +(`wait.go:24`), a test, and the literal `false` at `workflow_handler.go:845`. `AwakeableTimedOut` and +`AwakeableCancelled` are declared and **never assigned** in production code. A timed-out gate goes +through `handleMsgTimeout` (`:551`) into a generic `FunctionError{"execution timeout exceeded"}`, +indistinguishable from a crashed node, and the row stays `pending` forever. + +⚠️ **No external caller can learn a pending gate's awakeable id** (**C-07**). It exists only in a +journal entry's `Data` map; `SnapshotTimelineEvent` (`internal/workflow/execution_snapshot.go:47`) has +no `Data` field; `BuildTrace` handles only `step:*` and `state:changed` +(`internal/workflow/trace_builder.go:22-68`); snapshot and trace persist only from +`sendWorkflowCompleted` (`workflow_handler.go:518-519`), i.e. at terminal state; `GET /v1/workflows/{id}` +returns `dtos.GetWorkflowResponse{workflowId, status}` only; and +`AwakeableRepository.FindPending` (`internal/repositories/awakeable.go:16`) has **no production +caller** — only the two implementations and their tests. + +**C-05** (typed gate payloads: options, confirm, form) and **C-06** (durable timers and an expiry +sweeper — every timer today is a per-actor `SendAfter`, lost on restart) build on these. C-05's +"reject one item, keep the rest" semantics depend on A-11 being resolved, because per-item history +has nowhere to live. + +## § PROPOSED — B-04 MCP ingestion (nothing exists; no MCP reference anywhere in engine code) + +An MCP tool maps cleanly onto `ai.ToolDescriptor{FunctionID, MangledName, Description, Parameters}` +(`ai/tools.go:32`), and `ParameterSchemaToJSONSchema` (`:68`) already converts in the other +direction. **The blocker is execution, not description:** an ingested MCP tool has no code-backed +function, so `MapToRegistryPackage` registers it metadata-only with a nil `Transport`, and +`isExposableTool` rejects it — visible and un-callable until the remote transport (**B-02**) exists. +Name mangling (`MangleToolName`, `/` → `__`, `ai/tools.go:55`) also needs a rule for +server-qualified MCP names. + +**Ingestion beats exposure for now**: real customer estates already speak MCP, so importing their +tools is how they arrive on FUSE without a rewrite; exposing FUSE capabilities *as* an MCP server has +no customer behind it yet. Do not build both at once. + +## Traps + +| Trap | Evidence | +| --- | --- | +| Assuming an agent tool call is traced/journalled like a node call | The parity table above | +| Writing a registry predicate against `Metadata.Transport` | Hardcoded `Internal` at `loaded_package.go:70` | +| Assuming the agent transcript is durable | `messages`/`totalUsage`/`steps` are locals in `agent.go:146-148` | +| Assuming token cost is attributable to a run | `fuse_llm_*` labels are `{function, provider, model, type\|status}` only | +| Assuming `maxContextTokens` is a real token budget | `len(chars)/4`, `ai/context.go:13` | +| Assuming `outputSchema` enforces types | `valueMatchesType` accepts unknown type strings; a malformed schema silently disables the feature | +| Assuming an agent can wait, sleep or spawn a sub-workflow | `interceptedOrAsyncFunctionIDs`, `agent_tools.go:18-24` | +| Adding a sync, non-`CustomParameters` function anywhere | It is **automatically** offered to every agent as a tool; denylist it in the same PR | +| `AgentToolRegistry` cost at scale | `ListTools` rebuilds every descriptor and JSON schema on **every** agent execution; `InvokeTool` linear-scans `registry.List()` on **every** tool call | + +## Commands (confirmed to exist) + +```bash +make swagger # REQUIRED on a fresh clone before build/test +make lint && make build && make test # the mandated gate, in this order +go test ./internal/packages/functions/ai/... # agent, chat, tools, context, structured — pure unit tests, no actors +go test -run TestAgent ./internal/packages/functions/ai/ +go test ./internal/packages/ -run 'ListTools|InvokeTool|isExposableTool' +go test ./pkg/llm/... ./internal/llm/... +go test ./internal/workflow/ -run 'BuildTrace|BuildExecutionSnapshot|Journal' +make migrate # ./bin/fuse migrate — needed before any journal-enum change +make seed # ./bin/fuse seed examples -l debug +make infra-up # Postgres + S3 + etcd; engine on the host +LLM_OLLAMA_ENABLED=true LLM_OLLAMA_BASE_URL=http://localhost:11434/v1 \ + LLM_OLLAMA_MODEL=<model> LLM_DEFAULT_PROVIDER=ollama make run # the no-API-key path +curl -s localhost:9090/metrics | grep fuse_llm_ # the only cost surface that exists +``` + +⚠️ `./bin/fuse seed examples --ci` skips `github-request-example.json` and any file containing +`fuse/pkg/logic/timer` **or** `fuse/pkg/ai/` (`internal/app/cli/seed.go:184`), so the five `ai-*` +example schemas do not run in CI. There is no `make examples-ci` target; `--ci` is the only switch. + +$ARGUMENTS diff --git a/.agents/skills/crash-resume-testing/SKILL.md b/.agents/skills/crash-resume-testing/SKILL.md new file mode 100644 index 0000000..5e67dbd --- /dev/null +++ b/.agents/skills/crash-resume-testing/SKILL.md @@ -0,0 +1,314 @@ +--- +name: crash-resume-testing +description: Explains how to test FUSE durability for real — what the existing test tiers do and do not cover, how to stand up Postgres plus a filesystem object store, how to SIGKILL a real engine process at a reproducible point and assert on resume, and how to wire it into CI without slowing down `make test`. Use when building or extending the F-02 crash-resume harness, or when any change claims that a run survives a restart. +--- + +# crash-resume-testing + +`Workflow.Resume()` has exactly **one** call site (`internal/actors/workflow_handler.go:156`) and +**zero** test callers — `grep '\.Resume(' --include='*_test.go'` returns nothing. Everything FUSE +claims about durability rests on a code path no test has ever executed. This pack is the knowledge +base for F-02 in the repo-root backlog (`BACKLOG_V2.md`). *If the maintainer has renamed it to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog".* + +F-02 blocks all of Tier A, and the backlog is explicit that **several scenarios will be red and +that is the point — do not fix them here.** A red scenario gets a linked task, not a patch. + +## When to use + +- Building the F-02 harness, or adding a scenario to it. +- Any PR that claims a run survives a restart (A-02, A-03, A-05, A-07, C-06, C-08, C-12). +- Deciding where a durability regression test should live across the test tiers below. + +Read `.agents/skills/durable-execution-internals/SKILL.md` first — it explains what resume +actually does, which is what you are asserting against. + +## Ground truth + +| File | What it establishes | +| --- | --- | +| `internal/actors/workflow_handler.go:114-199` | `Init` — the trigger-vs-resume fork. The system under test. | +| `internal/workflow/workflow.go:187-306` | `Resume` / `replayJournalEntries` / `buildResumeAction` / `findPendingThreads`. | +| `tests/e2e/http.go`, `tests/e2e/disk.go` | Untagged helpers — polling, schema resolution. Reusable. | +| `tests/e2e/suite_e2e_test.go` | `RequireE2E`, `WorkflowsDirForTests`, `TestMain` and the `-workflows` flag. | +| `tests/e2e/workflow_fixture_e2e_test.go` | `UpsertSchema`, `TriggerExampleWorkflow`, `TriggerAndWaitTerminal`. | +| `docker-compose.yml` | The three profiles and the exact env each node gets. | +| `Makefile` | `test`, `test-functional`, `e2e-local`, `infra-up`, `ha-up`, `migrate`. | +| `.github/workflows/ci.yml`, `.../e2e.yml` | The `CI → E2E → CD` chain and where a new job attaches. | + +## The existing test surface, and its exact holes + +| Tier | Build tag | Command | Needs | Covers | Does **not** cover | +| --- | --- | --- | --- | --- | --- | +| Unit | *(none)* | `make test` (`./pkg/... ./internal/... ./tests/...`) | nothing | 100+ co-located `*_test.go`. Replay coverage is **two tests**: `TestReplayJournalEntries_SleepState` and `_CancelledState` (`internal/workflow/workflow_sleep_test.go:112,127`), which call `replayJournalEntries` directly and assert only the resulting `State`. `TestJournal_{Sleep,Awakeable,SubWorkflow}Entries` (`:18,41,64`) assert **write shape only**. | `Resume()` itself, `buildResumeAction`, `findPendingThreads`, any persistence round-trip. | +| Memory contract | *(none)* | `make test` | nothing | `tests/functional/*_test.go` untagged bodies — `contractTest<X>Repository` invoked as `TestMemory<X>Repository_Contract` (journal, workflow, graph, package, awakeable, environment, credential). | Postgres semantics. Memory `Append` never dedupes; Postgres has `UNIQUE (workflow_id, sequence)`. A replay bug that re-appends a sequence **passes silently here**. Also: **claims have no memory contract test** — `claim_repository_test.go` carries `//go:build functional`, so `make test` never exercises the claim contract (relevant to C-11). | +| Postgres contract | `functional` | `DB_POSTGRES_DSN=… make test-functional` | real PG | The same contract bodies against `postgres.New*Repository`, in schema `fuse_functional` (`tests/functional/postgres_test.go:22`). | Anything above the repository layer. No actor, no replay. **Skips silently** (`t.Skip`) when `DB_POSTGRES_DSN` is unset — green means nothing without the DSN. | +| E2E fast | `e2e` | `E2E_API_URL=… go test -tags=e2e ./tests/e2e` | running stack | `WorkflowResilienceSuite` — error edges, automatic retry, parallel retry, node timeout, all **in-process** failures with the engine alive throughout. `WorkflowPersistenceSuite` — trigger, finish, re-query, schema/package round-trip. | Process death. Nothing kills anything. The backlog names `workflow_resilience_suite_e2e_test.go` specifically: in-process failure is a **different thing** from crash-resume. | +| E2E slow | `e2e && e2e_slow` | `go test -tags="e2e e2e_slow" …` | running stack | `WorkflowOrchestrationSuite` — `sleep-test`, `subworkflow-test`, `awakeable-test`, `merge-strategy-test`, `timed-cond-test`. `WorkflowIntegrationSuite` — mermaid DAGs, `github-request-example`, `full-foundation-test`. | Restart. And note this tier **only runs on `main`** in CI (`e2e.yml`, `head_branch == 'main'`), so the durable primitives F-02 must kill mid-flight are today covered by a suite that never gates a PR. | + +**No example workflow uses `system/foreach`** — `grep -rl foreach examples/workflows/` returns +nothing (the `e2e/` overlay included). A ForEach scenario needs a new schema, and under Postgres it +is expected to fail before it ever reaches replay because the four `foreach:*` strings are missing +from the `journal_entry_type` ENUM (C-08). The ENUM gap is confirmed; the runtime failure is read +off the code, not executed — see the caveat under "Worth adding beyond the seven". + +## In-process failure vs real-process crash + +These are different systems under test and conflating them is the mistake F-02 exists to correct. + +| | In-process (what exists) | Real-process (what F-02 needs) | +| --- | --- | --- | +| What dies | one node function returns `FunctionError`, or a `SendAfter` timeout fires | the OS process | +| Actor tree | survives; the handler keeps its heap | destroyed — `forEachStates`, `iterThreadToForEach`, `RetryTracker`, every `SendAfter` timer, the OTel root span | +| Path exercised | `HandleNodeFailure` → retry / error edge | `WorkflowHandler.Init` resume branch → `LoadAll` → `LoadFrom` → `Resume` | +| Durable state | irrelevant | is the entire contract | +| Sufficient driver | memory | **Postgres + a shared object store only** | + +**A crash test under `DB_DRIVER=memory` is meaningless.** `MemoryWorkflowRepository.Get` returns +the live `*Workflow` pointer with populated threads/auditLog/aggregatedOutput; the Postgres driver +returns a fresh empty one rebuilt from the active graph definition. Resume behaves fundamentally +differently. Likewise `OBJECT_STORE_DRIVER=memory` dies with the process, so every +`input_ref`/`result_ref` fetch fails and `LoadAll` aborts — that is F-03's failure mode, not a +resume test. + +## Standing up the infrastructure + +Three compose profiles, one file. Confirmed properties that matter for a kill test: + +| Profile | Command | Nodes | `restart:` | Object store | +| --- | --- | --- | --- | --- | +| `infra` | `make infra-up` (`--profile infra up -d`) | none — engine runs on the host | n/a | rustfs S3 on `:9000` | +| `ha` | `make ha-up` (`--profile ha up --build -d`) | 3, built from source, `:9091-9093`, nginx LB `:9090` | **`unless-stopped`** — will fight a kill test | s3 | +| `e2e` | `docker compose --profile e2e up -d --wait` | 3, from `fuse-app:test`, `:9091-9093` | **`no`** — a killed container stays down until you start it | s3, bucket `fuse-e2e` | + +`infra` gives PG 17 on `:5432` (`fuse`/`fuse`/`fuse`), rustfs on `:9000` +(`rustfsadmin`/`rustfsadmin`), etcd on `:2379`. The `e2e` profile additionally runs `fuse-pg-init` +(`CREATE SCHEMA IF NOT EXISTS fuse_e2e`) and a dedicated `fuse-migrate` init container that the +nodes gate on with `service_completed_successfully` — the right shape when concurrent DDL matters, +since the `ha` profile has no migrate init container and every node runs `RunMigrations` itself at +boot (`internal/app/di/database.go:58`). + +**Recommended F-02 setup: `infra` profile + the engine as a host process.** You own the process +lifecycle, you can `kill -9` a PID with no container runtime in the way, and you can use the +filesystem object store so the payload graph is inspectable on disk. + +```bash +make infra-up +export DB_DRIVER=postgres +export DB_POSTGRES_DSN='postgres://fuse:fuse@localhost:5432/fuse?sslmode=disable&search_path=fuse_crash' +export OBJECT_STORE_DRIVER=filesystem +export OBJECT_STORE_FS_BASE_PATH=/tmp/fuse-crash-store +psql 'postgres://fuse:fuse@localhost:5432/fuse?sslmode=disable' -c 'CREATE SCHEMA IF NOT EXISTS fuse_crash' +make migrate # builds bin/fuse, then ./bin/fuse migrate (rebuild is part of the target) +./bin/fuse server -l debug -p 9090 & +``` + +Migrations honour whatever `search_path` the DSN carries — that is how `fuse_functional` and +`fuse_e2e` coexist in the same database. Give the crash harness its own schema so it does not +collide with either. Leave `HA_ENABLED` at its default `false` for the base scenarios: with HA off +the **only** recovery is the one-shot `RecoverWorkflows` message sent from `Fuse.Start` +(`internal/app/fuse.go:242`, send at `:248-249`) at boot, which is exactly the path you want to +assert on. Turn HA on only for a claim-specific +scenario, and then account for `HA_LEASE_TIMEOUT` (default 30s, 15s in the e2e profile): after a +SIGKILL `WorkflowClaimActor.Terminate` never runs, so `ReleaseWorkflows` is skipped and the dead +node's `claimed_by`/`claimed_at` stay on the row until the lease expires. + +## Killing the process deterministically + +`SIGKILL` only — `kill -9 <pid>`, or `docker kill --signal=KILL <container>`. `SIGTERM` runs the +graceful shutdown path (`SHUTDOWN_TIMEOUT`, `ReleaseWorkflows`, `Terminate` hooks) and tests +nothing. On restart, re-exec the same `bin/fuse server` with the same environment; the harness +asserts across the boundary. + +**Choose the kill point by observable state, never by `time.Sleep`.** Three reproducible levers, +best first: + +1. **Park the run in `sleeping`.** `system/sleep` and `system/wait` both call + `SetState(StateSleeping)` and then `persistWorkflowState()` **before** arming their `SendAfter` + (`workflow_handler.go:749-763`, `:765-795`). Poll `GET /v1/workflows/{id}` until it reads + `sleeping` and the journal is guaranteed flushed. `tests/e2e/http.go:189` + `WaitForWorkflowStatus(client, baseURL, wfID, "sleeping", timeout)` already does the polling. + This is the cleanest kill point in the engine. +2. **Poll the journal directly.** `SELECT entry_type FROM journal_entries WHERE workflow_id = $1` + until the specific `step:completed` you want to crash after appears. Every state-visible + transition is followed by `persistJournal`/`persistWorkflowState`, so a row in the table is a + hard happens-before edge. Use this for "kill after a completed step" and for the multi-thread + scenario (wait for exactly N `thread:finished` rows). +3. **Park on an external callback.** An async node returns `NewFunctionResultAsync()` and the run + waits for `POST /v1/workflows/{workflowID}/execs/{execID}`. Beware: the async result is + journaled as `step:completed` before the run parks (see Traps), so the DB status stays + `running` — poll for the `step:completed` row of that exec instead. + +Anti-pattern: killing after a fixed sleep. Runs finish in milliseconds under memory-free nodes, and +the whole test becomes a race against the engine. + +Do **not** reuse `TriggerAndWaitTerminal` (`workflow_fixture_e2e_test.go:30`) in a crash harness. +It silently re-triggers up to 3 times when a workflow is stuck in `"untriggered"` — a quarantined +HA flake workaround. In a crash test `"untriggered"` may be the thing under test, and a fresh +`workflowId` per retry destroys the assertion. + +## Assertion strategy + +Assert on the durable record, not on "it finished". A run that completes can still have corrupted +its journal on the way. + +| Property | How to assert | Guards | +| --- | --- | --- | +| **Journal length stable across three restarts** | `SELECT count(*) FROM journal_entries WHERE workflow_id = $1` before and after each restart, with the run parked and idle at each sample. Expect **equal**. | A-07 — replay's `SetResultFor` appends a fresh `step:completed` per previously-completed step, and `Init`'s `persistWorkflowState` flushes them. Growth is linear in restarts. | +| **No duplicate side effects** | `SELECT count(*) FROM sub_workflow_refs WHERE parent_workflow_id = $1` — expect exactly 1 per `system/subworkflow` node. `SELECT count(*) FROM awakeables WHERE workflow_id = $1` — expect 1 per gate. | A-02. Both are expected to be duplicated today — `handleSubWorkflowAction` mints a fresh `workflow.NewID()` and `handleSystemWait` a fresh `uuid.New()` on the re-dispatch path. **This is read off the code, not observed**; F-01.2 in the backlog exists to verify exactly the subworkflow half, and its "done when" allows for "it did not reproduce and why". | +| **Trace stability** | `GET /v1/workflows/{id}/trace` after terminal state; compare step count and per-step `duration`. | A-07's real symptom is not duplicated step rows (`BuildTrace` only creates one on `step:started`, which replay does not duplicate) but recomputed `CompletedAt`/`Duration` against the **replay** timestamp, and `Status` forced back to `completed`. | +| **Snapshot timeline stability** | `GET /v1/workflows/{id}/snapshot`, compare `timeline` length. | The timeline appends **every** entry (`execution_snapshot_builder.go:32`), so it grows by a full copy per restart. | +| **Sleep remainder honoured** | Record wall-clock before the kill; assert the run completes at roughly the original deadline, not `kill_time + full_duration`. | A-02 — `handleSystemSleep` re-parses `Args["duration"]` and re-arms the **full** duration. | +| **Pre-crash awakeable token still resolves** | Capture the awakeable id before the kill (today: from `journal_entries.data_ref` → `data.json`, or the `awakeables` table — there is no route, C-07); after restart `POST /v1/awakeables/{id}/resolve` and expect success. | This one is expected to **pass**: `resolve_awakeable.go:84-85` uses the awakeable's own stored `ExecID`/`ThreadID`, and `replayPendingThread` reuses the original execID. So the predicted failure is the **extra** row, not the old token — but that prediction is code reading, and this scenario is how you find out. | +| **Exactly one advance per gate** | After resolving, assert the thread advanced once — `count(*)` of `step:started` for the downstream node is 1. | A-02's double-advance. | + +**Exactly-once vs at-least-once — do not assert what has not been decided.** Today's behaviour is +at-least-once with **no dedup beneath it**: `replayPendingThread` re-issues a pending step with the +original execID and no journal marker, and +`internal/handlers/async_function_result.go` performs **no** pending-state or duplicate validation. +**A-03 is the open decision** — (a) at-least-once with a stable idempotency key the SDK dedupes on, +or (b) replay-aware re-arming that does not re-dispatch — and it must be settled in an ADR and +written into the B-01 protocol spec. Until then, write the assertion as *"the node was invoked N +times and the run's observable outcome is X"* and record N, rather than asserting "exactly once". +Making the harness encode a guarantee nobody has chosen is how the decision gets made by accident. + +## The seven F-02 scenarios + +Each is its own test. Expected-red is normal; link the owning task rather than fixing it. **The +"expected today" column is a prediction derived from reading the code — no scenario here has been +executed.** A scenario that comes out green is a real result: close the premise and say so, per the +backlog's "verify before fixing". + +| # | Scenario | Kill point | Assert | Expected today | +| --- | --- | --- | --- | --- | +| 1 | Kill after a completed step | poll `journal_entries` for the target `step:completed` | that node is not re-executed; journal has no second `step:started` for it | should pass, but journal grows by a duplicate `step:completed` → **A-07** | +| 2 | Kill with a pending async node | poll for the async exec's `step:completed` (it parks as completed) | the run resumes and the pre-crash callback URL still advances it exactly once | fragile — the parked step is **not** seen as pending, so resume does not re-arm it; the run waits on the external callback with no timer → informs **A-03** | +| 3 | Kill with a pending `system/subworkflow` | poll for `subworkflow:started` | **exactly one child after restart** — one row in `sub_workflow_refs`, one child run | predicted **red — A-02** (and this is F-01.2). A second child is minted at `workflow_handler.go:902`; the follow-on — both children completing and both notifying the same `ParentExecID` — is inference, and this scenario is the thing that settles it | +| 4 | Kill with a pending `system/sleep` | status `sleeping` | wake-up fires at the original deadline (remainder, not full duration) | **red — A-02** | +| 5 | Kill with a pending awakeable | status `sleeping` | pre-crash token still resolves; exactly one row in `awakeables`; one advance | token expected to resolve; predicted **red on the duplicate row and the reset timeout — A-02** (`handleSystemWait` re-mints at `workflow_handler.go:728`, re-arms the full timeout at `:789-794`) | +| 6 | Multi-thread run killed between thread completions | poll for exactly N `thread:finished` rows with no step in flight | every branch resumes | **premise is inference — C-12.** Loss is only on the zero-pending branch (`workflow.go:229-238`); the pending branch fans out correctly. `Next()` has side effects, so discarded actions have already journaled their `step:started` and the *next* restart picks them up — expect it to look flaky across one restart and self-heal across two. Verify before filing | +| 7 | Three restarts on one run | park, kill, restart, ×3 | journal length and `BuildTrace` output identical after each cycle | **red — A-07**, compounding | + +**Worth adding beyond the seven:** a run containing `system/foreach` under Postgres. The expectation +is that it fails before it ever reaches replay — the `foreach:*` types are absent from the +`journal_entry_type` ENUM (confirmed: 14 values across all 11 migrations, none of them `foreach:*`), +so the first `persistJournal` after the foreach node starts should fail the transaction and, because +`persistJournal` skips `MarkPersisted` on error (`workflow_handler.go:467-471`), the run's journal +freezes permanently. **That runtime consequence is derived from reading the code and has not been +executed against a live Postgres** — confirm it before spending time on ForEach replay logic (C-08). + +## Wiring it into CI without slowing `make test` + +**None of this exists yet — it is what F-02 has to add.** The only confirmed facts in this section +are the shape of the current pipeline; everything phrased as an instruction is a recommendation to +the implementer, not a description of the repo. + +The chain is `CI → E2E → CD`, each triggered by the previous workflow's `workflow_run` (`ci.yml` on +push/PR to `main`; `e2e.yml` on `workflows: ["CI"]`; `cd.yml` on `workflows: ["E2E"]`, +`branches: [main]`). Attach the harness to `.github/workflows/e2e.yml`, not to `ci.yml`. + +- **Give it its own build tag**, e.g. `//go:build e2e && e2e_crash`, mirroring the existing + `e2e && e2e_slow` pattern. Untagged files in `tests/e2e/` (`http.go`, `disk.go`, `doc.go`) stay + untagged so the helpers keep compiling under plain `go test ./tests/e2e`; only the test bodies + carry the tag. This keeps `make test` (which compiles `./tests/...` with **no** tags) completely + unaffected — it will pick up nothing new. +- **New job in `e2e.yml`**, modelled on `e2e-fast`: checkout `workflow_run.head_sha`, Go 1.26, + build `fuse-app:test`, bring up infra, run with an explicit `-timeout`, dump + `docker compose logs --tail=200` on failure, `down -v` in `always()`. +- **Gate it like `e2e-fast`, not like `e2e-slow`.** `e2e-slow` only runs when + `head_branch == 'main'`, i.e. after merge — a durability regression would land on main before + anyone saw it. F-02's whole value is catching it on the PR. +- **Budget the runtime.** `e2e-fast` uses `-timeout 10m` and `e2e-slow` `-timeout 15m` (`make + e2e-local` uses 15m). Seven kill cycles plus process restarts is minutes, not seconds; keep sleep + durations in the fixtures short and prefer journal polling over wall-clock waits. Note the + existing `sleep-test` sleeps **5s** in `examples/workflows/sleep-test.json` but **500ms** in the + e2e overlay `examples/workflows/e2e/sleep-test.json` — copy the overlay's magnitude, and be aware + a 500ms sleep is a poor kill window. +- **Do not add it to `scripts/pre-commit-gates.sh`.** That script runs `make lint && make build && + make test` and is the local gate; a Docker-dependent suite does not belong in it. +- `.github/workflows/e2e.yml` and `cd.yml` trigger on `workflow_run`, which means they execute from + the **default-branch** definition of the workflow file. Changes to `e2e.yml` on a PR branch are + not exercised by that PR — expect one merge before the new job actually runs. + +## Traps + +- **An async-parked step is journaled as `step:completed`.** `NewFunctionResultAsync()` carries + `Output.Status = FunctionSuccess` (`pkg/workflow/fn_result.go:43-49`) and + `handleMsgFunctionResult` calls `SetResultFor` **before** checking `Result.Async` + (`workflow_handler.go:275-281`). So `findPendingThreads` never sees an in-flight async node, and + the DB status stays `running`, not `sleeping`. Scenario 2 must poll the journal, not the status. +- **`t.Skip` makes an unconfigured suite green.** `tests/functional/postgres_test.go:testDSN` skips + without `DB_POSTGRES_DSN`; `RequireE2E` skips under `testing.Short()`. Assert in CI that the + expected number of tests actually **ran**, or a misconfigured job passes forever. +- **`E2E_API_URL` must be set.** `DefaultAPIURL` is `http://localhost:9090` (`http.go:15`) but the + e2e stack publishes `9091-9093`; the LB on `:9090` only exists in the `ha` profile. Every + make target and CI job sets `E2E_API_URL=http://localhost:9091` explicitly. +- **`examples/workflows/e2e/` silently shadows production schemas** of the same name during e2e + (`sleep-test`, `sum-rand-branch`, `timed-cond-test`, `timeout-test`, via + `ReadSchemaFileWithOverlay`). A local reproduction with the production JSON can behave + differently from CI. +- **A fresh clone cannot build.** `docs/docs.go` and `docs/swagger.{json,yaml}` are gitignored but + `internal/actors/mux_server.go` blank-imports the generated `docs` package. Run `make swagger` + before `make build`/`make test`. +- **Results are cached aggressively.** `go clean -testcache` before trusting a green replay run; + the e2e targets already pass `-count=1`. +- **Post-SIGKILL recovery is not instant under HA.** `ClaimWorkflows` will not take a row until + `claimed_at` is older than `HA_LEASE_TIMEOUT`, and the sweep claims a hardcoded batch of 10 per + `HA_CLAIM_SWEEP_INTERVAL` tick (`workflow_claim_actor.go:110`). Budget the timeout accordingly, + or run the base scenarios with `HA_ENABLED=false`. +- **`make test` runs without `-race`.** No gate enables it. Run `go test -race ./internal/... ./pkg/...` + by hand when a fix touches actors or the memory repositories. + +## Conventions to follow + +- Suite shape: a `testify` `suite.Suite` with `client *http.Client; baseURL string; workflowsDir string`, + a `func TestXxxSuite(t *testing.T) { t.Parallel(); suite.Run(t, new(XxxSuite)) }` entry point, and + a `SetupSuite` calling `RequireE2E(s.T())` + `WorkflowsDirForTests(s.T())` then looping + `UpsertSchema` over the schema ids. Crash suites should **not** be `t.Parallel()` against each + other if they share one engine process. +- Test method naming `TestThing_ObservableOutcome`, every assertion carrying a message that states + the expectation. Literal `// Arrange` / `// Act` / `// Assert` blocks, `require` for anything that + must stop the test and `assert` for the rest (`.agents/rules/07-testing.mdc`). +- Lint and the helper files: `.golangci.yml` declares `issues.exclude-rules` dropping `errcheck` and + `gosec` for `path: _test\.go`, but the untagged helpers in `tests/e2e/` (`http.go`, `disk.go`) are + **not** `_test.go` files, so both linters apply to them in full. Put helper code there in the same + shape they already use — `defer func() { _ = resp.Body.Close() }()` (`http.go:88,103,122`) and + `//nolint:gosec` on a deliberate path join (`disk.go:105`). (Whether the `_test.go` exclusion is + still honoured under golangci-lint v2's schema is unverified here — write test bodies defensively + and let `make lint` arbitrate.) +- The reproducing test lands in the same PR as the fix, **failing in the first commit** + (`BACKLOG_V2.md`, "Rules for the implementing agent"). One task, one PR. Bugs found along the way + get appended to the backlog with evidence, not fixed inline. +- Author guidance under `.agents/` only — `.claude/` and `.cursor/` are symlinks (ADR-0009). + +## Commands + +```bash +make swagger # first, on a fresh clone +make lint && make build && make test # the mandated gate, in this order +go test -v -run 'TestReplayJournalEntries' ./internal/workflow/ # the only existing replay tests +go clean -testcache + +make infra-up # PG 5432 + rustfs 9000 + etcd 2379 +make migrate # build + ./bin/fuse migrate (needs DB_POSTGRES_DSN) +DB_POSTGRES_DSN='postgres://fuse:fuse@localhost:5432/fuse?sslmode=disable' make test-functional + +make e2e-local # build image, --profile e2e up --wait, tags=e2e vs :9091, down -v +E2E_API_URL=http://localhost:9091 go test -tags=e2e ./tests/e2e -v -count=1 -timeout 10m +E2E_API_URL=http://localhost:9091 go test -tags="e2e e2e_slow" ./tests/e2e -v -count=1 -timeout 15m +docker compose --profile e2e logs --tail=200 && docker compose --profile e2e down -v + +make ha-up / make ha-down # 3 nodes from source; note restart: unless-stopped +``` + +## Related ADRs + +`docs/adr/0010-durable-execution-journal-and-replay.md` (the contract under test — note it +describes intent, not current behaviour), `0018-high-availability-and-clustering.md` (claims and +leases, and why a SIGKILL'd node's claim lingers), `0019-object-store-payload-externalization.md` +(why the object store must outlive the process), `0023-timeout-enforcement-model.md` (why every +deadline is an in-memory `SendAfter` and therefore dies with the process), +`0021-deployment-and-delivery-architecture.md` (the CI → E2E → CD chain and the compose profiles), +`0003-in-memory-repositories-by-default.md` (why a memory-driver green proves nothing here). + +$ARGUMENTS diff --git a/.agents/skills/durable-execution-internals/SKILL.md b/.agents/skills/durable-execution-internals/SKILL.md new file mode 100644 index 0000000..a06d92c --- /dev/null +++ b/.agents/skills/durable-execution-internals/SKILL.md @@ -0,0 +1,315 @@ +--- +name: durable-execution-internals +description: Maps the engine's durability machinery — the 18 journal entry types and which are actually replayed, the resume control flow from process start to a running run, the invariants a change must preserve, and the known defects with their backlog owners. Use when touching journal, replay, resume, persistence or payload externalization, or before starting any F/A-tier backlog task that claims a durability bug. +--- + +# durable-execution-internals + +The engine's durability story is "append-only journal + replay on recovery" (ADR-0010). The +journal defines **18** entry types. `Workflow.replayJournalEntries` handles **5**. That gap is the +single most important fact in this pack: most of what the journal records is written and never +read back, so several primitives silently do the wrong thing after a restart. Do not read a +write-only entry type as evidence that its behaviour is durable. + +## When to use + +- Changing anything under `internal/workflow/` (journal, replay, resume, threads, audit log, + projections) or `internal/actors/workflow_handler.go`. +- Changing `internal/repositories/postgres/journal.go` or `.../workflow.go`. +- Adding a journal entry type, or changing what an existing one carries. +- Starting F-01, F-02, A-02, A-03, A-05, A-07, A-08, A-09, A-11, C-06, C-08, C-12 from the + repo-root backlog (`BACKLOG_V2.md`). *If the maintainer has renamed it to `BACKLOG.md`, the + canonical file is the one titled "FUSE — product shape and backlog".* + +## Ground truth + +Read these before trusting any prose, including this file. + +| File | What it owns | +| --- | --- | +| `internal/workflow/journal.go` | The 18 type constants, `JournalEntry`, `Journal` (`Append`/`LoadFrom`/`NewEntries`/`MarkPersisted`). 136 lines, **zero read logic**. | +| `internal/workflow/workflow.go` | `Trigger`, `Resume`, `replayJournalEntries`, `buildResumeAction`, `findPendingThreads`, `replayPendingThread`, `Next`, `SetResultFor`, `HandleNodeFailure`, `RetryNode`. | +| `internal/actors/workflow_handler.go` | The actor owning one run: `Init` (trigger-vs-resume fork), system-function interception, all persistence calls. | +| `internal/repositories/postgres/journal.go` | Payload externalization (`input_ref`/`result_ref`/`data_ref`) and rehydration. | +| `internal/repositories/postgres/workflow.go` | The workflow envelope, `loadGraph`, `restoreState`. | +| `internal/repositories/postgres/migrations/000001_create_tables.up.sql` | The `journal_entry_type` ENUM and `journal_entries` schema. | +| `docs/adr/0010-durable-execution-journal-and-replay.md` | The intended contract. Useful as a spec to test against — **not** as a description of current behaviour (see Traps). | + +## The two-writer rule + +Journal writes are split across exactly two owners and the split is deliberate: + +- **The aggregate** (`internal/workflow/workflow.go` + `foreach.go`) writes the execution-graph + types via `w.journal.Append(...)`. +- **The actor** (`internal/actors/workflow_handler.go`) writes the system-primitive types by + reaching in via `a.workflow.Journal().Append(...)`. + +Keep new entry types on the correct side of that line. Aggregate state goes in the aggregate. +One exception already exists and is worth knowing about: `foreach:iteration:started` is a +system-primitive type written from the aggregate (`foreach.go:45`) because thread allocation +happens there — the other three `foreach:*` types are written by the actor. + +## Journal entry inventory + +`written` = the `Type:` field line of the `Append` call. `read` = a `case`/comparison that +structurally consumes the entry. Sites verified by grep in both directions. + +| Entry type | Written at | Read / replayed at | Meaning | +| --- | --- | --- | --- | +| `step:started` | `workflow.go:171` (Trigger), `:488` (RetryNode), `:616` (newRunFunctionAction), `foreach.go:51` | `workflow.go:207` (replay → `auditLog.NewEntry`), `:273` (findPendingThreads), `trace_builder.go:22`, `execution_snapshot_builder.go:55` | A node execution was dispatched. Carries `Input`. The **only** thing that creates a trace step row. | +| `step:completed` | `workflow.go:430-434` (`SetResultFor`, `Output.Status == FunctionSuccess`) | `workflow.go:209` (replay → `SetResultFor`), `:280`, `trace_builder.go:35`, `execution_snapshot_builder.go:79` | A node returned success. Carries `Result`. Replaying it is what rebuilds `aggregatedOutput`. | +| `step:failed` | `workflow.go:430-434` (`SetResultFor`, status != success) | `workflow.go:280` (findPendingThreads), `trace_builder.go:47`, `execution_snapshot_builder.go:90`, `internal/repositories/journal_memory.go:53` + `postgres/journal.go:252` (`FindFailed`) | A node returned an error. **Not replayed** — a failed step's `Result` is never restored into the audit log on resume. | +| `step:retrying` | `workflow.go:857` (`HandleNodeFailure`) | `trace_builder.go:62`, `execution_snapshot_builder.go:104` | An automatic retry was scheduled. **Not replayed** — which is exactly why `RetryTracker` is not rebuilt (**A-08**). | +| `step:manual-retry` | `workflow.go:478` (`RetryNode`), `Data.previousExecId` | **NEVER READ** in non-test code | Links a manual-retry exec to the exec it replaces. `SnapshotNodeRun.PreviousExecID` exists and is never populated. | +| `thread:created` | `workflow.go:166` (Trigger), `:603` (cross-thread `newRunFunctionAction`) | `workflow.go:204` (replay → `threads.New`), `execution_snapshot_builder.go:42` | A thread entered the registry. **The only way replay learns a thread exists.** | +| `thread:finished` | `workflow.go:318`, `:327`, `:348`, `:358`, `:371`, `:884` | `workflow.go:211` (replay), `execution_snapshot_builder.go:50` | A thread reached its terminal node. Replay collects these into `lastCompletedThreadIDs`. | +| `state:changed` | `workflow.go:550` (`SetState`) | `workflow.go:217` (replay, **direct field write**, bypassing `SetState`), `trace_builder.go:68`, `execution_snapshot_builder.go:112` | Run-level state transition. | +| `sleep:started` | `workflow_handler.go:752`, `Data{duration, reason}` | **NEVER READ — owned by A-02** | A `system/sleep` gate opened. | +| `sleep:completed` | `workflow_handler.go:815` | **NEVER READ — owned by A-02** | The sleep timer fired. | +| `awakeable:created` | `workflow_handler.go:782`, `Data{awakeableId, timeout}` | **NEVER READ — owned by A-02** | An external-callback gate opened. | +| `awakeable:resolved` | `workflow_handler.go:849` | **NEVER READ — owned by A-02** | An external caller resolved the gate. | +| `subworkflow:started` | `workflow_handler.go:918`, `Data{childWorkflowId, childSchemaId, async}` | **NEVER READ — owned by A-02** | A child run was spawned. | +| `subworkflow:completed` | `workflow_handler.go:1186` | **NEVER READ — owned by A-02** | A child run reported back. | +| `foreach:started` | `workflow_handler.go:991` | **NEVER READ — owned by C-08** | A `system/foreach` began; `Data` carries the batch counts. | +| `foreach:iteration:started` | `foreach.go:45` | **NEVER READ — owned by C-08** | One iteration thread was allocated. **Note: no `thread:created` is written alongside it.** | +| `foreach:iteration:completed` | `workflow_handler.go:1071` | **NEVER READ — owned by C-08** | One iteration finished. | +| `foreach:completed` | `workflow_handler.go:1082` | **NEVER READ — owned by C-08** | All batches finished. | + +**Partial-read caveat.** The ten "NEVER READ" types (and `step:manual-retry`, which is likewise +consumed by nothing) *are* appended to `ExecutionSnapshot.Timeline` by +`execution_snapshot_builder.go:32-39`, which records Sequence/Timestamp/Type/ThreadID/ExecID/NodeID +and **drops `Data`**. They are visible as opaque timeline events; nothing structural consumes them. + +**Postgres ENUM gap.** The schema half is confirmed: `journal_entry_type` has 13 values from +migration `000001` plus `step:manual-retry` from `000004` = **14**, no migration anywhere adds a +`foreach:*` value, the Go file defines 18, and `postgres/journal.go:84` inserts `string(entry.Type)` +straight into that column. The runtime consequence below is **read off the code, not executed +against a live Postgres — verify it before acting on it**: under `DB_DRIVER=postgres` the first +`persistJournal` after a `system/foreach` node starts should fail the whole transaction, and since +`persistJournal` skips `MarkPersisted` on error (`workflow_handler.go:467-471`) the same poisoned +batch is retried and fails forever, freezing that run's journal at the last pre-foreach sequence. +Any C-08 work starts with an `ALTER TYPE ... ADD VALUE` migration (pattern: +`migrations/000004_add_manual_retry_journal_type.up.sql`) or it cannot be observed at all. + +## Resume control flow, hop by hop + +**Non-HA boot recovery** (the only recovery path when `HA_ENABLED=false` — it is a one-shot, there +is no periodic retry): + +1. `internal/app/fuse.go:242` `Fuse.Start` → `node.Send` to `workflow_sup` with + `{Type: RecoverWorkflows}` (`fuse.go:248-249`; `WorkflowSupervisorName == "workflow_sup"`). +2. `internal/actors/workflow_sup.go:212` `recoverWorkflows` → `FindByState(untriggered, running, sleeping)` (`:215`). +3. Per row: `workflowRepository.Get(id)` → `spawnWorkflowActor(schemaID, wf.ID(), wf.Environment())` (called at `workflow_sup.go:233`, declared at `:239`) → `StartChild(WorkflowInstanceSupervisor, ...)`. + +**HA recovery**: `internal/actors/workflow_claim_actor.go:108` `sweep` → +`ClaimWorkflows(nodeID, 10)` (`postgres/claim.go:24`, `FOR UPDATE SKIP LOCKED`, states +untriggered/running/sleeping, steals only when `claimed_at < NOW() - HA_LEASE_TIMEOUT`) → one +`TriggerWorkflow` message per claimed row → same `spawnWorkflowActor`. The actor is only registered +when `HA_ENABLED=true` (`fuse.go:217`), and the only claim repository that does anything is the +Postgres one, selected only when `DB_DRIVER=postgres` **and** a pool exists +(`internal/app/di/repos.go:82-89`; otherwise `NewMemoryClaimRepository()`, logged as "no-op"). +So in practice this path needs both. + +**The resume path itself** — `internal/actors/workflow_handler.go:114` `Init`: + +| Hop | Line | What happens | +| --- | --- | --- | +| 1 | `:129` | `workflowRepository.Exists` → true takes the resume branch. | +| 2 | `:130` | `workflowRepository.Get`. **Postgres** (`postgres/workflow.go:45`) builds a *fresh* `workflow.New` with empty threads/auditLog/aggregatedOutput, resolving the graph via `loadGraph(ctx, schemaID)` — the **active** definition, no version predicate (**A-05**). **Memory** (`internal/repositories/workflow_memory.go:39`) returns the *live pointer*. | +| 3 | `:133` | `claimForThisNode` (`:366`) — on loss it sends `NewWorkflowCompletedMessage(wfID, "claimed-elsewhere")` to `a.Parent()` and returns `false`, leaving persisted state untouched. Returns `true` on a claim-store **error** — "fails open", stated in its own doc comment at `:360-365` (**C-11**). | +| 4 | `:138-139` | `a.workflow.SetSecretResolver(a.newSecretResolver(a.workflow.Environment()))`, `graphService.EnsureNodeMetadata`. | +| 5 | `:143` | `startRootSpan`. Note `startWorkflowTimeout` is **not** called here — it is only called at `:193` on the fresh-create path, so a schema's total timeout is lost after any restart. | +| 6 | `:145` | `State() == StateUntriggered` ⇒ `Trigger()` instead of replay. | +| 7 | `:150` | `journalRepo.LoadAll`. On error it **logs and continues without `LoadFrom`** — see Traps. | +| 8 | `:154` | `Journal().LoadFrom(entries)` — sets `seq` to the last entry's `Sequence` and `lastPersisted = seq`. | +| 9 | `:156` | `workflow.Resume()`. | +| 10 | `:159` | `handleWorkflowAction(action)`. | +| 11 | `:163` | `persistWorkflowState()` = `persistJournal()` + `workflowRepository.Save`. **This is what flushes replay's duplicate appends (A-07).** | + +**Inside `Resume()`** (`workflow.go:187`): + +- Empty journal ⇒ `NoopAction`. Otherwise `replayJournalEntries` then `buildResumeAction`. +- `replayJournalEntries` (`:199-223`) iterates in sequence order and handles five types: + `thread:created` → `threads.New` (which **resets** the thread to running, `thread.go:47`); + `step:started` → `auditLog.NewEntry`; `step:completed` → **`SetResultFor`**, which sets the audit + result, does `aggregatedOutput.Set(nodeID, ...)` **and appends a new journal entry**; + `thread:finished` → `threads.Get(...).SetState(ThreadFinished)` + collect the id; + `state:changed` → direct write to `w.state.currentState`. The other 13 fall through. +- `buildResumeAction` (`:226-257`): `findPendingThreads` builds `started[execID]` from + `step:started` and `completed[execID]` from `step:completed|step:failed`, returning every + started-but-not-completed exec. + - **0 pending** ⇒ loop `lastCompletedThreadIDs` calling `Next(threadID)` and return the **first** + non-noop action, discarding the rest (**C-12**). + - **1 pending** ⇒ `replayPendingThread`. + - **>1 pending** ⇒ `RunParallelFunctionsAction` over all of them (this branch fans out correctly). +- `replayPendingThread` (`:295-306`) returns `RunFunctionAction{FunctionExecID: workflow.ExecID(pt.execID)}` + — the **original** execID from the journal, reused, and **no journal entry is appended**. + +**Re-dispatch.** The action goes back through `handleWorkflowAction` → `handleWorkflowRunFunctionAction` +(`workflow_handler.go:671`), whose switch at `:675-688` re-intercepts `system/sleep`, +`system/wait`, `system/subworkflow`, `system/foreach` — which is precisely how A-02's duplicate +side effects happen. + +## Invariants any change must preserve + +1. **Sequence numbers are assigned solely by `Journal.Append`** (`journal.go:81`), which also + stamps `entry.Timestamp = time.Now()` unconditionally. There is no way to append with a + caller-supplied sequence or timestamp. Any replay-time append therefore carries a *replay-time* + timestamp — plan around it, do not fight it in the projections. +2. **`lastPersisted` is a watermark, not content dedup.** `persistJournal` flushes `NewEntries()` + (everything with `Sequence > lastPersisted`) and only then calls `MarkPersisted`; it + deliberately does **not** mark on error (`workflow_handler.go:467-471`). One poisoned entry + blocks every subsequent entry for that run forever. +3. **Postgres enforces `UNIQUE (workflow_id, sequence)`** (`idx_journal_wf_seq`). The memory + driver does not dedup at all. A replay bug that re-appends an existing sequence errors loudly + under Postgres and passes silently under memory. +4. **Replayed steps must be idempotent.** `replayPendingThread` re-issues the original action with + no marker, so a re-dispatched node runs again with the same execID. Nothing beneath it dedupes + (**C-10**). +5. **Payload refs must resolve.** `postgres/journal.go:160-206` fetches every non-nil + `input_ref`/`result_ref`/`data_ref` through an errgroup with `SetLimit(10)`; a single failed + object fetch aborts the whole `LoadAll`. Keys are deterministic and sequence-scoped: + `workflows/{workflowID}/journal/{sequence}/{input|result|data}.json` (`journal.go:28`). +6. **Every entry type string must exist in both the Go constants and the Postgres ENUM.** Adding a + Go constant without a migration is a runtime insert failure, not a compile error. Enum + migrations are effectively irreversible (`000004`'s down file is a comment explaining why). +7. **Reconstruction must not journal.** `postgres/workflow.go:380` `restoreState` calls `SetState` + (which appends `state:changed` at sequence 1) and relies on `LoadFrom` to discard it. A second + reconstruction path must honour the same contract. +8. **Projections are pure.** `BuildTrace` and `BuildExecutionSnapshot` both say so in their doc + comments. New read logic takes `[]JournalEntry` and returns a value. The one place this is + violated today is `replayJournalEntries` calling the mutator `SetResultFor`. + +## Known defects — do not mistake these for the contract + +| Backlog id | Defect | Evidence | +| --- | --- | --- | +| **A-02** | Restart with a pending `system/subworkflow` mints a **second child** — `findPendingThreads` matches the `step:started` (`subworkflow:started` is never read), `replayPendingThread` re-issues, the switch re-intercepts, and `handleSubWorkflowAction` calls `workflow.NewID()` at `workflow_handler.go:902`. The duplicate child is what the backlog asserts. The follow-on — the first child row still being `running`/`sleeping` so recovery re-drives it too, and both children notifying the same `ParentExecID` and double-advancing the parent — is **inference from the control flow, not an observed run**; F-02 scenario 3 is what confirms it. | `workflow.go:267-306`, `workflow_handler.go:682-684`, `:901-933` | +| **A-02** | Restart with a pending `system/wait` calls `uuid.New()` again (`workflow_handler.go:728`), saving an **additional** awakeable row and appending an additional `awakeable:created`, and re-arms the timeout with the **full** original duration (`:789-794`). Both rows carry the same `ExecID`, so resolving either advances the run and resolving both double-advances it. *Note: the pre-crash token still resolves correctly, because `resolve_awakeable.go` looks up the awakeable's own stored `ExecID`/`ThreadID` and `replayPendingThread` reuses the original execID.* | `workflow_handler.go:727-747`, `internal/handlers/resolve_awakeable.go` | +| **A-02** | Restart with a pending `system/sleep` re-parses `Args["duration"]` from the journaled input and `SendAfter`s the **full** duration (`workflow_handler.go:710-763`). A 24h sleep with 10 minutes left restarts at 24h. The remainder is computable — `sleep:started`'s `Timestamp` and `Data["duration"]` are both persisted. | `workflow_handler.go:710-763` | +| **A-07** | **Duplicate journal append on replay — compounds every restart.** `LoadFrom` sets `lastPersisted = seq`; replay's `case JournalStepCompleted` calls `SetResultFor`, which unconditionally appends at `workflow.go:434` with a *fresh* sequence; `Init`'s `persistWorkflowState` (`:163`) flushes them. Each restart adds one duplicate `step:completed` per previously-completed step. **Constraint for the fix:** `SetResultFor`'s `aggregatedOutput.Set` (`:428`) is the only thing that rebuilds the data plane during replay — gate the `Append` at `:434` only; skipping the call empties aggregated output. | `journal.go:100-108`, `workflow.go:209`, `:428-440` | +| **A-07** | Trace corruption is real but not "each step N+1 times". `BuildTrace` only creates a step row on `step:started` (`trace_builder.go:22`), which replay does **not** duplicate. A duplicate `step:completed` mutates the existing row in place: `CompletedAt`/`Duration` are recomputed against the *replay* timestamp (`trace_builder.go:37-41`), so after a restart six hours later every completed step reports a ~6h duration, and `Status` is forced back to `completed` even for steps that ended `retrying`/`failed`. The snapshot **timeline** does grow by a full copy per restart (`execution_snapshot_builder.go:32`). | as cited | +| **A-09** | **Payload write amplification.** `postgres/journal.go:46-68` does one unconditional `putJSON` per non-nil `Input`, `Result` and `Data` — no threshold, no inlining, no compression, no dedup — and keys are sequence-scoped, so an identical payload at two sequences is stored twice. `postgres/workflow.go:81-97` re-PUTs the entire aggregated output to `workflows/{id}/output.json` on **every** `Save`, which `persistWorkflowState` calls on every state transition (`workflow_handler.go:392-397` = `persistJournal` + `Save`). `postgres/trace.go:75,83` adds two more PUTs per step per save. `LoadAll` re-fetches the whole *payload* graph — every `input_ref`/`result_ref`/`data_ref` object — on every resume. | as cited | +| **A-11** | **`aggregatedOutput` is keyed by node ID, not exec ID** — `workflow.go:428` is `w.aggregatedOutput.Set(entry.FunctionNodeID, result.Output.Data)`. A node that runs twice destroys its own history in the data plane; the journal keeps every execution, `SourceFlow` only ever sees the latest (read side: `applyFlowMapping`'s `aggregatedOutput.Get(mapping.Variable)`). This is the promoted Tier-A design risk. **Do not start by re-keying the KV** — the backlog wants an ADR first, and note `pkg/store/kv.go` is objx-backed with dot-notation, so a composite key needs a separator that is not `.`, and `SetResultFor` is also the replay rebuild path so any re-key changes replay semantics at the same time. | `workflow.go:428`, `pkg/store/kv.go:71-84` | +| **C-08** | **ForEach state does not survive restart.** `forEachStates map[string]*ForEachState` and `iterThreadToForEach map[uint16]string` (`workflow_handler.go:97-100`) are re-initialised to empty maps in `Init` (`:118-119`) and never reconstructed, although all four `foreach:*` types are written. Worse: `StartForEachIteration` (`foreach.go:42-51`) creates the dynamic thread with `threads.New` but journals `foreach:iteration:started`, **not `thread:created`** — so on resume the iteration thread does not exist in `w.threads` while its `step:started` still yields a pending thread; `Next` then does `w.threads.Get(threadID)` followed immediately by `currentThread.CurrentExecID()` with no nil check (`workflow.go:310-311`) — a nil-pointer panic. | as cited | +| **C-08** | **Interception bypass.** `spawnForEachBatch` (`workflow_handler.go:1043-1046`) sends the `RunFunctionAction` straight to `WorkflowFuncPoolName`, so the switch at `:675-688` never runs and `system/*` inside a ForEach body reaches the pool as an ordinary function (the implementations in `internal/packages/functions/system/` are success-returning placeholders). **A second, unlisted instance of the same bypass:** the `ActionRetryFunction` branch at `workflow_handler.go:658-668` also `SendAfter`s directly to the pool. | as cited | +| **C-12** | **Multi-thread resume — premise is inference, verify before fixing.** The "first non-noop wins" loss is real but only on the **zero-pending** branch (`workflow.go:229-238`); the pending branch fans out correctly. The reproducing shape is "crashed after several threads recorded `thread:finished` with no step in flight". Be warned: `Next()` has side effects — the discarded calls have **already** journaled their `step:started`, so the next restart picks them up as pending. Expect it to look flaky across one restart and self-heal across two. | `workflow.go:226-257` | + +## Traps — looks right, is wrong + +- **An async-parked step is journaled as `step:completed`, not left pending.** + `NewFunctionResultAsync()` sets `Output.Status = FunctionSuccess` (`pkg/workflow/fn_result.go:43-49`), + and `handleMsgFunctionResult` calls `SetResultFor` **before** checking `Result.Async` + (`workflow_handler.go:275-281`). So `findPendingThreads` does **not** see an in-flight async node + as pending, and there is no journal marker distinguishing "awaiting a callback" from "done". + A-03 is the open decision on this: (a) at-least-once with a stable idempotency key the SDK + dedupes on, or (b) replay-aware re-arming of the original execID. The backlog leans toward (a) + ("the SDK contract makes (a) far more palatable"). *This pack's reading, not the backlog's text:* + (b) would need a new "parked" entry type to tell the two states apart, and therefore an enum + migration. Do not assume the pending-step machinery covers async. +- **`LoadAll` failure is silent and then corrupting.** `Init:150-155` logs and proceeds *without* + `LoadFrom`. The journal still holds the sequence-1 `state:changed` that `restoreState` appended, + `lastPersisted` is 0, `Resume()` sees a non-empty journal and returns `Noop`, and the trailing + `persistWorkflowState` tries to insert sequence 1 again — violating `idx_journal_wf_seq`. This is + exactly the F-03 combination (postgres journal + memory object store), and + `internal/app/di/objectstore.go:38` falls through to memory for **any** unrecognized driver string. +- **Never validate a resume fix against the memory driver.** `MemoryWorkflowRepository.Get` returns + the live `*Workflow` with populated threads/auditLog/aggregatedOutput; Postgres returns a fresh + empty one. Replay under memory stomps live thread state (`threads.New` replaces the thread with a + fresh running one) and re-registers audit entries on top of an already-populated aggregate. A + green memory test proves nothing about Postgres. +- **`threads.AllFinished()` returns `true` for an empty thread map** (`thread.go:84-95`). A resumed + run whose threads were not reconstructed can be declared complete by `checkWorkflowCompletion` + while work is outstanding. +- **Replay never journals its own re-dispatch.** After N restarts a node that ran N+1 times still + shows exactly one `step:started`. Per-attempt accounting (A-08, cost attribution) cannot be built + from the journal as it stands. +- **ExecIDs are UUIDv8 with the thread id embedded** (`pkg/workflow/exec_id.go:13,23`) and only 12 + bits are decoded (low nibble of byte 6 + byte 7), capping thread ids at 4095. Async callbacks + recover their thread purely by parsing the execID with no lookup. Never mint an execID with a + plain `uuid.New()` for anything that returns through `handleMsgAsyncFunctionResult`. +- **A node ID containing `.` breaks the data plane silently.** `aggregatedOutput` is objx-backed + with dot-notation paths, so `Set("a.b", …)` writes a nested path, not a flat key. +- **ADR-0010 overstates current behaviour.** Its L34-37 enumerates all 18 types as "covering the + full lifecycle" and L40-43 says `Resume()` "replays entries to rebuild threads, audit log, and + aggregated output" — true only for the five handled types. If A-02 lands, ADR-0010 needs an + amendment or a superseding ADR, not a silent code change. +- **`GET /v1/workflows/{workflowID}/status` does not exist**, whatever older prose says. + The route table (`internal/actors/mux_worker.go`) registers, for a run, + `/v1/workflows/{workflowID}` (`:220`, returns `{workflowId, status}` only — `get_workflow.go:49-64`), + `/cancel` (`:230`), `/snapshot` (`:240`), `/retry-node` (`:250`), `/retry` (`:260`), + `/v1/awakeables/{awakeableID}/resolve` (`:270`), `/trace` (`:280`) and + `/v1/workflows/{workflowID}/execs/{execID}` (`:79`). Trust `mux_worker.go`. + +## Inspecting a run's durable state locally + +```bash +make infra-up # docker compose --profile infra up -d (PG 5432, rustfs S3 9000, etcd 2379) +export DB_POSTGRES_DSN='postgres://fuse:fuse@localhost:5432/fuse?sslmode=disable' +make migrate # builds bin/fuse, then ./bin/fuse migrate +make run # ./bin/fuse server -o -p 9090 -l debug (set DB_DRIVER/OBJECT_STORE_DRIVER first; + # note -o also starts the ergo observer app — drop it for a plain engine) +``` + +Relevant tables (all created by the embedded migrations): +`workflows` (`workflow_id, schema_id, state, output_ref, snapshot_ref, environment, claimed_by, claimed_at` +— **no version column**, A-05), `journal_entries`, `awakeables`, `sub_workflow_refs`, +`execution_traces`, `execution_trace_steps`, `graph_schemas`, `graph_schema_versions`, +`node_heartbeats`, `idempotency_keys`. + +```bash +psql "$DB_POSTGRES_DSN" -c "SELECT sequence, entry_type, thread_id, exec_id, function_node_id + FROM journal_entries WHERE workflow_id = '<id>' ORDER BY sequence;" +psql "$DB_POSTGRES_DSN" -c "SELECT unnest(enum_range(NULL::journal_entry_type));" # expect 14, no foreach:* +psql "$DB_POSTGRES_DSN" -c "SELECT state, claimed_by, claimed_at FROM workflows WHERE workflow_id = '<id>';" +``` + +Duplicate `step:completed` rows for one `exec_id` at widely separated sequences are the A-07 +signature: one extra pair per restart. + +Object-store layout (key builders live at the top of each postgres repo file): + +| Key | Built by | +| --- | --- | +| `workflows/{id}/journal/{seq}/{input,result,data}.json` | `postgres/journal.go:28` | +| `workflows/{id}/output.json` | `postgres/workflow.go:28` | +| `workflows/{id}/execution-snapshot.json` | `workflow_handler.go:414` (bypasses the repository layer) | +| `workflows/{id}/trace/{execID}/{input,output}.json` | `postgres/trace.go:29` | +| `awakeables/{id}/result.json` | `postgres/awakeable.go:29` | +| `schemas/{id}/definition.json`, `schemas/{id}/v{n}/definition.json` | `postgres/graph.go:32`, `:16` | + +With `OBJECT_STORE_DRIVER=filesystem` these are plain files under `OBJECT_STORE_FS_BASE_PATH` +(default `./data/fuse`) — the easiest layout to inspect while debugging replay. + +HTTP read surface: `GET /v1/workflows/{id}/snapshot` serves the persisted snapshot if +`snapshot_ref` is set, otherwise builds one **live** from `journalRepo.LoadAll` + +`BuildExecutionSnapshot` (`internal/handlers/get_workflow_snapshot.go:59-98`) — the closest thing +to a mid-flight read model that exists today, though its `aggregatedOutputs` comes from a +repo-reconstructed `Workflow` and is therefore empty under Postgres (C-09). +`GET /v1/workflows/{id}/trace` reads `traceRepo.FindByWorkflowID` only and 404s until the run +terminates — snapshot and trace are persisted solely from `sendWorkflowCompleted` +(`workflow_handler.go:518-519`). + +## Related ADRs + +| ADR | Title | Why it matters here | +| --- | --- | --- | +| 0010 | Durable execution via an append-only journal | The governing decision. A change to replay semantics amends or supersedes it. | +| 0011 | Thread model for fork/join and ForEach iteration | Dynamic thread allocation, the ForEach limitation. | +| 0018 | High availability: claims, clustering, and schema replication | Who re-drives a crashed run, and the lease semantics. | +| 0019 | Externalize large payloads to a pluggable object store | The `*_ref` contract, and its accepted "no GC, orphan objects" cost. | +| 0020 | Observability: Prometheus metrics, OpenTelemetry tracing, execution traces | What the trace projection is for. | +| 0022 | Retry and error-handling model | `HandleNodeFailure`, `RetryTracker`, error edges. | +| 0023 | Timeout enforcement via actor timers | Why every deadline is in-memory `SendAfter` and dies with the process (C-06). | +| 0032 | Sub-workflow composition | `SubWorkflowRef` and parent notification. | +| 0003 | In-memory repositories by default, Postgres optional | Why two implementations exist and must be kept behaviourally identical. | + +Coding rules that bind changes here: `.agents/rules/03-actor-patterns.mdc`, +`05-repositories.mdc`, `07-testing.mdc`, `12-quality-gates.mdc` (all `alwaysApply`). Author +guidance under `.agents/` only — `.claude/` and `.cursor/` are symlinks (ADR-0009). + +Quality gate, in this order: `make lint && make build && make test`. For anything touching the +journal schema, also `DB_POSTGRES_DSN=... make test-functional` — the memory driver will not catch +a sequence collision or an ENUM violation. + +$ARGUMENTS diff --git a/.agents/skills/function-package-authoring/SKILL.md b/.agents/skills/function-package-authoring/SKILL.md new file mode 100644 index 0000000..6ed756a --- /dev/null +++ b/.agents/skills/function-package-authoring/SKILL.md @@ -0,0 +1,259 @@ +--- +name: function-package-authoring +description: Explains how an in-process FUSE node function is declared, registered, executed and tested — file layout, metadata, the FunctionResult contract including the async Finish/callback path, what ExecutionInfo actually carries, and the registration traps that silently produce a non-executable node. Use when adding or changing anything under internal/packages/functions/, when a node fails with "function X has no transport", or when a backlog task touches node dispatch. +--- + +# function-package-authoring + +This pack covers the **in-process half** of node execution: a Go function compiled into the engine, +registered from code, dispatched to a `WorkflowFuncPool` worker. + +The **out-of-process half** — a node backed by someone else's HTTP service, the invocation envelope, +ack semantics and the callback contract — is the remote node protocol. **It does not exist yet**: +**B-01** (write the spec) and **B-02** (implement the transport in Core) are both open backlog tasks, +and the inventory of what is actually shipped lives in the `remote-node-protocol` pack +(`.agents/skills/remote-node-protocol/SKILL.md`). Do not design a remote transport from this file; +point at that one. What this file gives B-02 is the seam it must plug into: +`transport.FunctionTransport`, `MapToRegistryPackage`'s branch, and the async completion route. + +Backlog ids below are from the repo-root backlog (`BACKLOG_V2.md`). *If it has been renamed to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog".* + +## When to use + +- Adding a function to an existing package, or adding a whole package. +- Diagnosing "function %s not found" / "function %s has no transport". +- Any task that reads or changes how a node's result reaches the workflow actor. + +## Ground truth + +| File | What it is authoritative for | +| --- | --- | +| `pkg/workflow/function.go` | The entire function contract: `type Function func(*ExecutionInfo) (FunctionResult, error)` | +| `pkg/workflow/execution_info.go` | Exactly what a function receives (5 fields, lines 14–23) | +| `pkg/workflow/fn_result.go` | The five result constructors — there are no others | +| `pkg/workflow/metadata.go` | The declarative metadata types an author fills in | +| `pkg/workflow/package.go` | `NewPackage` (L29), `NewFunction` (L69), `PackagedFunction.Function` is `json:"-"` (L24) | +| `internal/packages/loaded_package.go` | Registration-time mapping + the executable/metadata-only branch (L112–128) | +| `internal/packages/internal_packages.go` | `List()` (L41) — the one hardcoded slice of compiled-in packages | +| `internal/actors/workflow_func.go` | The worker that actually calls the function | +| `docs/adr/0024-package-registry-and-function-metadata.md` | Why the registry is metadata-described | + +⚠️ **`.agents/rules/04-workflow-nodes.mdc` is wrong about this area and is `alwaysApply: true`.** Its +`FunctionMetadata` / `InputMetadata` / `OutputMetadata` samples invent fields that do not exist +(`FunctionMetadata.ID/Name/Description`, `InputMetadata.Required`, `OutputMetadata.Schema`) and omit +`Transport` and all edge metadata; the file is also duplicated verbatim (689 lines, body repeats at +L348). Trust `pkg/workflow/metadata.go`, not the rule. + +## The contract, in one screen + +```go +// pkg/workflow/function.go +type Function func(*ExecutionInfo) (FunctionResult, error) +``` + +`ExecutionInfo` — `pkg/workflow/execution_info.go:14-23`. **This is all a function gets today:** + +| Field | Type | Notes | +| --- | --- | --- | +| `WorkflowID` | `workflow.ID` | run id | +| `ExecID` | `workflow.ExecID` | UUIDv8 with the thread id in bytes 6–7; `ExecID.Thread()` recovers it | +| `Environment` | `string` | ADR-0031 resolution scope; scope data, not a secret | +| `Input` | `*FunctionInput` | typed accessors over the mapped input KV | +| `Finish` | `func(FunctionOutput)` | **nil until a transport binds it** (`internal/packages/transport/internal.go:47`) | + +There is **no `context.Context`**, no cancellation, no logger, and no graph node id. +`internal/actors/workflow_func.go:96` is literally `_ = nodeCtx` — the node span context is created +and discarded. **A-04** owns adding the context; **B-02** depends on it (a remote call cannot inject +`traceparent` without one). See the `observability-tracing` pack. + +## File layout — copy an existing package + +Every package is one directory `internal/packages/functions/<pkg>/` with `package.go` plus one file +per function named after the function. Confirmed for `debug`, `logic`, `http` and `system`. `ai` +follows it for its two *registered* functions (`chat.go`, `agent.go` — `package.go` registers only +`ChatFunctionID` and `AgentFunctionID`) and additionally carries helper files that are not functions +(`context.go`, `structured.go`, `tools.go`, `usage.go`). + +``` +internal/packages/functions/logic/ + package.go // doc comment + `const PackageID` + `New() *workflow.Package` + sum.go // SumFunctionID + SumFunctionMetadata() + SumFunction() + sum_test.go +``` + +Each function file exports exactly three things: + +```go +const SumFunctionID = "sum" // no slash, ever +func SumFunctionMetadata() workflow.FunctionMetadata { ... } // a func, never a package-level var +func SumFunction(execInfo *workflow.ExecutionInfo) (workflow.FunctionResult, error) { ... } +``` + +Worked end to end: `internal/packages/functions/logic/sum.go` (57 lines) — metadata declares one +required input `values []float64` and one output `sum`; the body reads +`execInfo.Input.GetFloat64SliceOrDefault("values", ...)` and returns +`workflow.NewFunctionResult(workflow.FunctionSuccess, map[string]any{"sum": sum})`. + +Package ids are namespaced `fuse/pkg/<domain>` (`fuse/pkg/http`, `fuse/pkg/logic`, `fuse/pkg/debug`, +`fuse/pkg/ai`). `system` is the one deliberate exception — `internal/packages/functions/system/package.go:9` +sets `PackageID = "system"` because its functions are engine-intercepted. Do not "fix" it; the +package id is derived by splitting the schema's `function` string at the **last** slash +(`internal/services/graph_service.go:314`), so every existing schema depends on it. + +## Registration: two paths, only one produces an executable node + +| Path | Entry point | Result | +| --- | --- | --- | +| **Code-backed internal** | `internal_packages.go:List()` → `PackageService.RegisterInternalPackages` → `Registry.Register` → `MapToRegistryPackage` | `NewLoadedInternalFunction` wires `InternalFunctionTransport`; the node runs | +| **API / persistence** | `PUT /v1/packages/{packageID:.+}` → `dtos.FromPackageDTO` → `PackageService.Save` → `Registry.Register` | metadata-only, `Transport == nil`; dispatch errors with `function %s has no transport` (`loaded_package.go:41`) | + +The branch is `internal/packages/loaded_package.go:112`: +`if function.Metadata.Transport == transport.Internal && function.Function != nil`. Anything else +falls to the metadata-only `NewLoadedFunction` at L124. **B-02** is the task that makes the else-branch +produce a real remote transport. + +Adding a package is a **3-file minimum** change: + +1. the new `internal/packages/functions/<pkg>/` directory; +2. append it to `internal/packages/internal_packages.go:List()` (L41) — **omit this and the package does + not exist at runtime**, with no error anywhere; +3. only if it needs dependencies: widen `NewInternal` (L25) and the providers in + `internal/app/di/di.go` `PackageModule` (which provides `NewPackageRegistry`, `NewInternal`, + `providePackageRegistration`). + +Dependencies use a closure factory: `makeChatFunction(providers, usage) workflow.Function` +(`ai/chat.go:94`), with the ports declared *inside* `functions/<pkg>` and adapted in +`internal/packages` (`usage_recorder.go`, `agent_tools.go`) so `functions/<pkg>` never imports +`internal/packages` — that would be an import cycle. + +## Metadata and schemas + +`workflow.FunctionMetadata{Transport, Input, Output}`. Slice-shaped here; converted to the map-shaped +`internal/packages.FunctionMetadata` at registration. + +- `Transport: transport.Internal` — imported from `github.com/open-source-cloud/fuse/internal/packages/transport`. +- `Input.Parameters []ParameterSchema` — `{Name, Type, Required, Validations, Description, Default}`. +- `Input.CustomParameters: true` means schemaless input read straight from `Input.Raw()` (see `logic/if.go`). +- `Output.ConditionalOutput` + `ConditionalOutputField` + named `Output.Edges` with + `ConditionalEdge.Value` drive conditional routing (`logic/if.go` declares `if-true` / `if-false`). +- Empty slices are written `make([]T, 0)`, not nil — dominant convention (`sum.go:29` + `ParameterSchema`, `sum.go:42` `OutputEdgeMetadata`, `timer.go:33,37-38`). `system/sleep.go`, + `wait.go` and `subworkflow.go` omit `Output.Edges` entirely, while `system/foreach.go:42` declares + two; both shapes compile. + +## FunctionResult: the five constructors, and only five + +| Constructor | `Async` | Use | +| --- | --- | --- | +| `NewFunctionResult(status, data)` | false | general sync result | +| `NewFunctionResultSuccess()` | false | success, nil data | +| `NewFunctionResultSuccessWith(data)` | false | success with data | +| `NewFunctionResultError(err)` | false | **returns `(result, nil)`** — logical failure is a `FunctionError` result, not a Go error | +| `NewFunctionResultAsync()` | true | "I will call `Finish` later" | + +There is no `NewFunctionResultAsyncWith`. The Go `error` return is reserved for *unexpected* execution +failures; `workflow_func.go:141-153` rewrites a non-nil error into a `FunctionError` result anyway and +logs a warning when you return both, so returning both is redundant. + +## The async path + +**In-process (TODAY, works):** return `NewFunctionResultAsync()` synchronously *and* spawn a goroutine +that eventually calls `execInfo.Finish(...)`. Smallest reference: `logic/timer.go:44-70` (70 lines). +Richest reference: `ai/chat.go:94-174` (`makeChatFunction`) — every error path calls +`execInfo.Finish(workflow.NewFunctionOutput(workflow.FunctionError, map[string]any{"error": err.Error()}))` +before returning. + +`Finish` is bound by `internal/packages/transport/internal.go:47`; it sends +`messaging.NewAsyncFunctionResultMessage` to the handler addressed by +`gen.Atom(actornames.WorkflowHandlerName(wfID))` — by **name, not PID** (see the comment at L12-17; +sending by captured PID from a pool worker has been observed to fail with `gen.ErrUnsupported`). +It lands at `workflow_handler.go:handleMsgAsyncFunctionResult` (L313). + +**External callback (TODAY, exists, unvalidated) — described in ONE place, and it is not this file.** +The route is `POST /v1/workflows/{workflowID}/execs/{execID}` +(`internal/actors/mux_worker.go:79`), handled by `internal/handlers/async_function_result.go`. It is +the seam B-02 builds on, and it validates nothing today. Do **not** restate its contract from here: +the body DTO, the exact missing checks, the duplicate-callback consequence and the fix template all +live in `.agents/skills/remote-node-protocol/SKILL.md` § TODAY, which is the single source for +anything crossing the engine↔worker boundary. This pack owns only the **in-process** `Finish` path +above. + +**Why async is the default for anything slow (A-10).** `internal/actors/workflow_func_pool.go:40` is +`act.PoolOptions{PoolSize: 3}` — three workers **per workflow instance**, hardcoded. A synchronous node +holds one for its whole duration, so one 90-second call caps that run at two other concurrent branches. +`fuse/pkg/http/request` (`internal/packages/functions/http/request.go:128`) is synchronous and blocks +this way today. A-10 makes the size configurable and requires the remote transport never to hold a +worker while awaiting a callback. + +## How a schema references a function + +`examples/workflows/github-request-example.json` is the worked example. A node names the **full** id: + +```json +{ "id": "github-http-request", "function": "fuse/pkg/http/request" } +``` + +Edge `input` entries feed the declared parameters and read the declared outputs: + +- `{"source": "schema", "value": "https://api.github.com", "mapTo": "host"}` — literal from the schema. +- `{"source": "flow", "variable": "github-http-request.status", "mapTo": "statusCode"}` — a prior node's + output, addressed **by node id**. (Addressing by execution is **A-11**, and it is not built.) + +Binding happens at upsert time in `internal/services/graph_service.go:populateNodeMetadata` (L295): +split at the last slash → `Registry.Get(pkgID)` → `LoadedPackage.GetFunctionMetadata(node.Function)` +with the **full** id → `graph.UpdateNodeMetadata`. A schema naming an unregistered function fails +`PUT /v1/schemas/{schemaID}` and fails `fuse seed examples`. + +If a new example cannot run in CI, extend `shouldSkipExampleWorkflow` +(`internal/app/cli/seed.go:184`) — it currently skips `github-request-example.json` and any file whose +JSON contains `fuse/pkg/logic/timer` or `fuse/pkg/ai/`. Give a new example a **unique schema id**: files +are seeded in sorted-name order and the last file wins for a duplicated id. + +## Tests + +| Kind | Template | Package | +| --- | --- | --- | +| Function behaviour | `internal/packages/functions/logic/sum_test.go` | external (`package logic_test`); hand-builds `&workflow.ExecutionInfo{WorkflowID, ExecID, Input, Finish: nil}` | +| Metadata / package shape | `internal/packages/functions/system/package_test.go` | same package; asserts function count, parameter names, `Required`, and the `Full*FunctionID` constants | +| Registry / transport | `internal/packages/loaded_package_test.go` | `TestExecuteFunction_PersistenceDecodedInternalFunc_DoesNotPanic` (L33), `TestExecuteFunction_CodeBackedInternalFunc_Runs` (L60), `TestRegister_DataOnlyCopy_DoesNotDowngradeExecutableFunc` (L83) — uses `Encode`/`DecodePackage` to simulate a persistence round-trip | + +⚠️ `internal/packages/functions/http/` contains **only** `package.go` and `request.go` — zero test files. +Anything B-02 asserts about HTTP nodes is new test surface, including the synchronous baseline. + +## Traps (all confirmed in this tree) + +| Trap | Evidence | +| --- | --- | +| **Two packages named `transport`.** Metadata must import `internal/packages/transport` (defines `Internal`). `pkg/transport` defines only `HTTP`/`gRPC` — and `gRPC` is lowercase, so unexported and unusable outside that package. Importing the wrong one compiles into a metadata-only registration that silently never executes. | `pkg/transport/type.go` (11 lines); `internal/packages/transport/type.go` | +| **`Functions` is keyed by the FULL id.** `GetFunctionMetadata` and `ExecuteFunction` both expect `"fuse/pkg/http/request"` despite the parameter being named `functionID`. Passing `"request"` returns `function request not found`. | `loaded_package.go:66`, `:26`, `:35` | +| **A function id must not contain a slash.** `graph_service.go:314` splits at the last slash, so `fuse/pkg/http/request_async` resolves to package `fuse/pkg/http`, function `request_async`. | `graph_service.go:314` | +| **`PackagedFunction.Function` is `json:"-"`.** Any package arriving as JSON (API, Postgres reload, replication) loses its code pointer. `registry.go:47-65` exists solely to stop that downgrading an executable registration — it preserves the whole previously-registered `*LoadedFunction`. Do not "simplify" it. | commit 802675d | +| **The REST package DTOs drop all edge metadata**, both directions. `internal/dtos/package.go` has no `Edges` field anywhere, so `GET /v1/packages` shows `fuse/pkg/logic/if` without its `if-true`/`if-false` edges, and an API-registered package cannot declare conditional routing. It bites in-process authoring only when a package round-trips through the API; the registration-loss inventory and its backlog routing are owned by `.agents/skills/remote-node-protocol/SKILL.md`. | grep `Edges` in `internal/dtos/package.go` → no hits | +| **Adding a sync, non-`CustomParameters` function silently exposes it to every `ai/agent` as an LLM tool.** If it has side effects, add its full id to `interceptedOrAsyncFunctionIDs` in the same PR. | `agent_tools.go:18-24`, `isExposableTool` L95-111 | +| **The registry is a process-global singleton** (`var pkgRegistry Registry`, `registry.go:28`); `NewPackageRegistry` returns the same instance every call. Tests that register leak across tests in the same binary — use `MapToRegistryPackage` directly, as `loaded_package_test.go` does. | `registry.go:28-38` | +| **`FunctionMetadata.Concurrency` and `.RateLimit` are never populated.** `MapToRegistryPackage` does not set them and `pkg/workflow.FunctionMetadata` has no such fields, so the guards at `workflow_func.go:102`/`:108` are unreachable for code-backed packages. **No backlog task owns this** — file it, don't fix it in passing. | `internal/packages/function_metadata.go:14-15` vs `loaded_package.go:63-131` | +| **Intercepted `system/*` functions never reach a worker** — `handleWorkflowRunFunctionAction` (L671) switches on `SleepFullFunctionID`/`WaitFullFunctionID`/`SubWorkflowFullFunctionID`/`ForEachFullFunctionID` at L675-688 and returns. Their bodies are placeholders that return success (`system/wait.go:32`). Their **replay** is broken and is **A-02**. | `workflow_handler.go:675-688` | +| **ForEach bypasses that interception.** `spawnForEachBatch` builds an `ExecuteFunctionMessage` and sends straight to the pool (`workflow_handler.go:1043-1046`), so `system/*` inside a ForEach body hits the no-op placeholder and silently does nothing. Owned by **C-08**. | `workflow_handler.go:1013-1048` | +| **`http.ErrMethodNotAllowed` is declared and never used** (grep: two hits, both the declaration); the method is not checked against an allowlist. | `request.go:22-23` | + +## Commands (confirmed to exist) + +```bash +make lint && make build && make test # the mandated gate, in this order +go test -v ./internal/packages/... # registry, mapping, agent-tool exposure +go test -v -run TestSumFunction ./internal/packages/functions/logic/ +go test -v ./pkg/workflow/... # input / result / exec-id contract +make swagger # REQUIRED on a fresh clone before build/test +make build && ./bin/fuse seed examples -l debug # `make seed`; upserts examples/workflows/*.json +./bin/fuse seed examples --ci # applies shouldSkipExampleWorkflow +make run # ./bin/fuse server on :9090 +curl -s http://localhost:9090/v1/packages | jq '.items[].id' # dtos.PackageListResponse{metadata, items} +curl -X POST "http://localhost:9090/v1/workflows/$WF/execs/$EXEC" \ + -H 'Content-Type: application/json' -d '{"result":{"status":"success","data":{}}}' +``` + +Note: there is no `make examples-ci` target and no `scripts/run-example-workflows.sh` in this tree. +The real entry points are `make seed` and the `--ci` flag. + +$ARGUMENTS diff --git a/.agents/skills/observability-tracing/SKILL.md b/.agents/skills/observability-tracing/SKILL.md new file mode 100644 index 0000000..274af30 --- /dev/null +++ b/.agents/skills/observability-tracing/SKILL.md @@ -0,0 +1,287 @@ +--- +name: observability-tracing +description: Explains FUSE's three observability layers — OTel spans, Prometheus metrics, and the persisted execution trace — with the exact span names, attribute conventions and carrier helpers in use, the three trace-context breaks A-04 owns with file:line, and how to verify a trace end to end against a real collector. Use when touching internal/tracing, internal/metrics, internal/logging, span or metric call sites, or when working A-04 or C-04. +--- + +# observability-tracing + +Backlog ids below are from the repo-root backlog (`BACKLOG_V2.md`). *If it has been renamed to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog".* + +## When to use + +- Adding or moving a span, an attribute, a metric, or a log call. +- A-04 (trace context end to end) and C-04 (GenAI telemetry and cost). +- Any change that crosses an actor boundary and should stay in one trace. + +## Ground truth + +| File | Authoritative for | +| --- | --- | +| `internal/tracing/provider.go` | The **entire** OTel surface — 115 lines, `NewProvider` + 4 exported methods | +| `internal/actors/workflow_handler.go` | The `workflow.execute` root span, every `InjectCarrier` call site, the workflow counters, `persistTrace` | +| `internal/actors/workflow_func.go` | The `node.execute` child span and the discarded context (L96) | +| `internal/messaging/message.go:39-45` | `Message.TraceCarrier` — the only trace-context transport in the system | +| `internal/metrics/registry.go` | The whole `fuse_*` metric set, in a dedicated registry | +| `internal/metrics/ergo_collector.go` | The `ergo_*` gauges over `gen.Node.Info()` | +| `internal/workflow/trace.go`, `trace_builder.go` | The persisted `ExecutionTrace` (the *other* notion of "trace") | +| `internal/logging/app_logger.go` | The global `logFormat` every other logger reads | +| `docs/adr/0020-observability-metrics-tracing-execution-traces.md` | Why three layers, and that the two "trace" notions overlap | +| `docs/adr/0029-llm-cost-and-usage-tracking-and-budgets.md` | Phase A (usage metrics) shipped; budgets deferred | + +## ADR-0020's three layers, and the naming collision + +1. **Prometheus metrics** — always on, scraped at `GET /metrics`. +2. **OTel spans** — optional (`OTEL_ENABLED`, default `false`), needs a collector. +3. **Persisted `ExecutionTrace`** — always available, queryable over REST, built from the journal. + +(2) and (3) are both called "trace" and are unrelated. Nothing links them: migration +`000006_create_execution_traces` has **no `trace_id` or `span_id` column**, and +`internal/workflow/trace.go` has no such field. Given an `ExecutionTrace` there is no way to find the +corresponding OTel trace. If A-04 or C-04 wants that link it is a new column and a new migration. + +## Provider setup + +`tracing.NewProvider(cfg)` is provided by `CommonModule` (`internal/app/di/di.go`) and shut down from +`internal/app/fuse.go` (`(*Fuse).Terminate` → `tracingProvider.Shutdown`). + +```go +// internal/tracing/provider.go — the complete exported surface +func (p *Provider) StartSpan(ctx, name string, attrs ...attribute.KeyValue) (context.Context, trace.Span) // L90 +func (p *Provider) InjectCarrier(ctx) map[string]string // L95 +func (p *Provider) ExtractCarrier(ctx, carrier map[string]string) context.Context // L102 — early-returns ctx when len==0 +func (p *Provider) Shutdown(ctx) error // L110 — nil-tp safe +``` + +Enabled path (L34-80): `otlptracegrpc` exporter, `semconv v1.30.0` ServiceName/ServiceVersion, a +composite `TraceContext{} + Baggage{}` propagator, and `otel.SetTracerProvider`/`SetTextMapPropagator`. +Config: `OTEL_ENABLED` (default `false`), `OTEL_EXPORTER_OTLP_ENDPOINT` (default `localhost:4317`), +`OTEL_SERVICE_NAME` (`fuse`), `OTEL_SERVICE_VERSION` (`unknown`), `OTEL_EXPORTER_OTLP_INSECURE` (`true`). + +**Access tracing only through `*tracing.Provider`.** No code in the repo calls `otel.Tracer(...)` or +`otel.GetTextMapPropagator()` directly. It is a concrete struct, not an interface, so it cannot be +faked — unit tests must build a real provider from a config. + +## Conventions in use (this is the complete list) + +**Span names — exactly two:** + +| Name | Created at | Ended at | +| --- | --- | --- | +| `workflow.execute` | `workflow_handler.go:213-221` `startRootSpan()`, called from L143 (existing-workflow / replay branch) and L189 (fresh create) | `sendWorkflowCompleted()` L511 | +| `node.execute` | `workflow_func.go:90-95` | L113, L131, L166 (three exit paths) | + +**Attribute keys — exactly six**, dotted lowercase, always `attribute.String`: +`workflow.id`, `workflow.schema_id` (root span, L217-218), `workflow.final_state` (L510), +`node.exec_id`, `node.function_id`, `node.package_id` (`workflow_func.go:92-94`; the node span also +repeats `workflow.id` at L91). One event exists: +`nodeSpan.AddEvent("node.error", …)` at `workflow_func.go:160`. + +Status/error idiom: `span.SetStatus(codes.Ok, "")` / `codes.Error` plus `span.RecordError(err)` — +see `workflow_func.go:111-113, 129-131, 157-166`. + +**Trace context crosses actor boundaries as `map[string]string`, never as a `context.Context`** (ergo +messages are values): + +```go +// sender internal/actors/workflow_handler.go:699 +messaging.NewExecuteFunctionMessage(id, execAction, env, a.tracingProvider.InjectCarrier(a.spanCtx)) +// receiver internal/actors/workflow_func.go:89 +parentCtx := a.tracingProvider.ExtractCarrier(context.Background(), msg.TraceCarrier) +``` + +`NewExecuteFunctionMessage` is the **only** message constructor that accepts a carrier. Adding one +elsewhere means adding the parameter to that constructor in `internal/messaging/`. + +## The three A-04 breaks, with exact locations + +### Break 1 — inbound extraction is missing at the HTTP trigger + +`internal/handlers/trigger_workflow.go:HandlePost` (L65) has the `*http.Request` in hand and **never +reads `r.Header`** (grep confirms zero `r.Header` references in that file). It sends at L98 via +`messaging.NewTriggerWorkflowWithEnvMessage(schemaID, workflowID, environment)` — +`internal/messaging/trigger_workflow.go:30`, which takes no carrier, so `Message.TraceCarrier` is nil. +`workflow_sup.go` reads only `msg.TriggerWorkflowMessage()` (case at L98, read at L99) and calls +`spawnWorkflowActor(schemaID, workflowID, environment)` at L118 (declared at L239). The run's root span is then started +from `context.Background()` (`workflow_handler.go:214-216`) — a brand-new trace, disconnected from the +caller. + +`ExtractCarrier` has exactly **one** production call site in the whole repo (`workflow_func.go:89`), +so it is genuinely never called at any HTTP boundary. + +**Every path that starts a run has this break — there are seven, not four:** + +| Entry point | Send site | +| --- | --- | +| HTTP trigger | `internal/handlers/trigger_workflow.go:98` | +| Webhook | `internal/handlers/webhook.go:97` | +| Retry from scratch | `internal/handlers/retry_workflow.go:94` | +| Event bus | `internal/actors/event_trigger.go:103` | +| Cron | `internal/actors/cron_scheduler.go:95` (no inbound parent — this is the one that needs a *documented* root) | +| HA claim sweep | `internal/actors/workflow_claim_actor.go:121` | +| Sub-workflow spawn | `internal/actors/workflow_handler.go:929` — already has a live parent in `a.spanCtx`, so it is the cheapest to parent correctly | + +⚠️ Threading a carrier through the spawn chain touches +`internal/actors/workflow_instance_sup.go:56` (`if len(args) != 3`) and +`WorkflowHandlerInitArgs` (`workflow_handler.go:104`), which currently carries schemaID/workflowID/environment. + +⚠️ `ExtractCarrier` accepts only `map[string]string`; there is **no** `http.Header` helper. Do not +convert `r.Header` naively — W3C keys are lowercase (`traceparent`, `tracestate`) while Go +canonicalises header names, and `propagation.MapCarrier` is a plain map lookup. Add a header-aware +helper on `Provider` (wrapping `propagation.HeaderCarrier`) rather than converting at each call site. + +### Break 2 — handler → worker already works; it needs a regression test + +Inject at `workflow_handler.go:699` (dispatch), `:663` (retry, via `SendAfter`) and `:1044` +(`spawnForEachBatch`) → `messaging.Message.TraceCarrier` → extract at `workflow_func.go:89`. +**Leave the mechanism alone.** There is no test of it today: `internal/actors` contains only +`event_trigger_test.go`, `event_trigger_dedup_test.go`, `execution_timer_test.go`, `mux_server_test.go`. + +⚠️ **A regression test written against the default provider will pass while the code is broken.** +`noopProvider()` (`provider.go:82-87`) sets `prop: propagation.NewCompositeTextMapPropagator()` — a +composite with **zero** propagators — so with `OTEL_ENABLED=false` `InjectCarrier` returns an empty map +and `ExtractCarrier` returns the context unchanged. `internal/tracing/provider_test.go:49-61` admits +this in a comment ("A noop context produces an empty carrier"). A real test must build a provider with +`Otel.Enabled=true`, or otherwise install a real `propagation.TraceContext`. + +### Break 3 — the node span context is discarded + +`internal/actors/workflow_func.go:96` is exactly `_ = nodeCtx`. `nodeCtx` is bound at L90 by +`StartSpan(parentCtx, "node.execute", …)`; `nodeSpan` **is** used (L111-113, L129-131, L159-166) — only +the context is thrown away. It appears exactly once in the repo. + +`pkg/workflow/execution_info.go:14-23` has no `context.Context` field, so node code cannot create a +child span or inject `traceparent`. Confirmed: the literal string `traceparent` appears **nowhere** in +the Go tree. `internal/packages/functions/http/request.go` builds headers solely from the node's +`headers` input, and `pkg/http/client.go` creates its own `context.Background()` with a timeout — +there is no caller-context seam at all. `go.mod` carries no `otelhttp`/contrib package, so inbound and +outbound propagation must be hand-rolled or a dependency added. + +### Why A-04 is critical rather than nice-to-have + +This is the seam **every SDK crosses**. B-02 (implement the remote transport) must inject +`traceparent` into the outbound invocation envelope specified by B-01, and it cannot: there is no +context to inject from. A distributed engine whose selling point is traceability, that breaks the +trace at exactly the boundary it invites you to extend across, is unsellable. A-04's third item — +"add the context to `ExecutionInfo`" — is therefore a **public `pkg/` API change**, which per the +backlog's rules needs a changelog entry and a migration path. + +⚠️ **Tension to resolve in the A-04 ADR, not silently.** +`docs/adr/0027-async-tool-invocation-sub-execution-channel.md` (Status: Proposed) states explicitly +that per-execution runtime capability *must not* be added back as a field on +`workflow.ExecutionInfo` — a prior `Handle any` field was removed for exactly that reason. A-04 wants +a context field there. Address the conflict in the ADR. + +## Metrics surface + +`internal/metrics/registry.go` — a **dedicated** `prometheus.Registry`, never the global default. +A new series is a field on `FuseMetrics`, constructed in `NewFuseMetrics`, and appended to the +`reg.MustRegister(...)` list. + +| Series | Type | Labels | +| --- | --- | --- | +| `fuse_workflows_active` | Gauge | — | +| `fuse_workflows_completed_total` / `_failed_total` / `_cancelled_total` | Counter | — | +| `fuse_node_exec_duration_seconds` | HistogramVec (DefBuckets) | `function_id`, `status` | +| `fuse_llm_tokens_total` | CounterVec | `function`, `provider`, `model`, `type` | +| `fuse_llm_calls_total` | CounterVec | `function`, `provider`, `model`, `status` | +| `ergo_*` (uptime, processes, memory, …) | Collector over `gen.Node.Info()` | `node` | + +Scrape path: `mux_server.go` registers the ergo collector into the Fuse registry, then serves +`promhttp.HandlerFor(prometheus.Gatherers{fuseRegistry, combinedRegistry}, …)` at `GET /metrics` +(the combined registry adds the Go and process collectors). No handler is wrapped in tracing middleware. + +**Packages that must not import prometheus declare a narrow port instead.** +`internal/packages/functions/ai/usage.go` defines `UsageRecorder` (`RecordUsage`, `RecordCall`) plus +`NopUsageRecorder`; `internal/packages/usage_recorder.go` is the metrics-backed adapter. Follow that +pattern for any C-04 recorder that must reach spans from the `ai` package. + +## C-04 — what exists, what is missing, and the open question + +**Exists (TODAY):** the two span names and six attributes above; LLM usage as Prometheus counters +only, via the `UsageRecorder` port, recorded from `ai/chat.go`, `ai/agent.go` and `ai/structured.go`. + +**Missing (PROPOSED, C-04):** agent id, model, tokens in/out, cost, gate decision, tenant. Concrete +blockers per attribute: + +| Wanted | Blocker | +| --- | --- | +| graph node id | `messaging.ExecuteFunctionMessage` has no node id field. It is recoverable handler-side via `a.workflow.AuditLog().Get(execID.String()).FunctionNodeID` — the pattern already used at `workflow_handler.go:693` to look up the execution timeout — so adding a field to the message is the mechanical route. | +| model, tokens, agent id | need a span reachable from `internal/packages/functions/ai`, which today has **no** tracing dependency at all. Depends on A-04 break 3. | +| cost | **no data source.** `pkg/llm` `Usage` carries prompt/completion/total tokens only; ADR-0029 defers the pricing table. | +| tenant | no tenancy concept exists in Core (it is Tier D, `D-01`). | + +**The open question C-04 must answer first: which attributes belong to the engine, and which to the +caller/gateway.** The backlog's own guidance is "follow OTel GenAI conventions where they exist; +namespace the rest" and "meter cost at the gateway" — and the model gateway is **C-03**, which C-04 +depends on. Cost per se cannot be computed in Core today. Decide the split in an ADR before writing +code; do not let C-04 silently absorb the adjacent undecided items (OTel sampling and +`TraceRetentionConfig`, declared at `internal/workflow/trace.go:37` and referenced nowhere else in +the Go tree — grep returns only the type and its doc comment, so nothing enforces it. ADR-0020's +consequences flag both as not yet centrally tuned, at `docs/adr/0020-…:58`). + +## Verifying a trace end to end (not a unit test) + +A unit test cannot prove propagation — see the noop-provider warning above. A-04's "done when" +requires a real collector, and **no collector exists anywhere in this repo**: grepping +`docker-compose.yml`, `deploy/` and `.env.example` for `otel` returns nothing, and no file sets +`OTEL_ENABLED`. Standing one up is part of the task; `docker-compose.yml` already uses profiles +(`infra`, `ha`, `e2e`), so a collector service belongs there. + +Procedure: + +1. Start a collector (or Jaeger/Tempo) exposing OTLP/gRPC on `:4317`. +2. `make build && OTEL_ENABLED=true OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 \ + OTEL_EXPORTER_OTLP_INSECURE=true OTEL_SERVICE_NAME=fuse ./bin/fuse server -l debug` +3. Trigger with an inbound parent — **this is the break-1 repro; today the header is read by nothing**: + ```bash + curl -s -X POST http://localhost:9090/v1/workflows/trigger \ + -H 'Content-Type: application/json' \ + -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' \ + -d '{"schemaID":"smallest-test"}' + ``` +4. In the collector, assert one trace containing `workflow.execute` with `node.execute` children — and, + after A-04, that its trace id is `4bf92f3577b34da6a3ce929d0e0e4736`. + +⚠️ The e2e helpers cannot send custom headers: `tests/e2e/http.go` builds the request internally and +takes no header argument — `POSTJSON` (L112) sets only `Content-Type`, `GET` (L94) sets none at all. +An inbound-traceparent e2e test needs a new helper there. + +## Traps (all confirmed) + +| Trap | Evidence | +| --- | --- | +| **Cancelled workflows leak the span and the gauge and skip the trace.** `handleMsgCancelWorkflow` (L579) sets `StateCancelled` and returns without calling `sendWorkflowCompleted()`, whose only two callers are L480 and L486. So `rootSpan.End()` (L511) never runs, `WorkflowsActive.Dec()` (L498) never runs, the `StateCancelled` branch (L506-508) is unreachable from cancellation so `fuse_workflows_cancelled_total` stays 0, and `persistTrace()` never runs. **Do not fix this inside an A-04 PR** — the backlog forbids drive-by fixes; file it with evidence. | `workflow_handler.go:480, 486, 494-519, 579` | +| **Intercepted `system/*` functions produce no `node.execute` span and no duration metric.** `handleWorkflowRunFunctionAction` returns at L675-688 before dispatch, so sleeps, waits, sub-workflow spawns and foreach fan-outs are invisible in OTel and contribute nothing to `fuse_node_exec_duration_seconds`. | `workflow_handler.go:671-688` | +| **Async node spans measure dispatch, not work.** `nodeSpan.End()` (L166) fires as soon as `ExecuteFunction` returns, and every `ai/chat`, `ai/agent` and `logic/timer` node returns `NewFunctionResultAsync()` immediately. The completion path (`transport/internal.go` `sendAsyncFunctionResult`, and `handlers/async_function_result.go`) carries **no** `TraceCarrier` at all. | `workflow_func.go:166`; `transport/internal.go:18-25` | +| **Replay/recovery mints a second, unrelated trace.** `startRootSpan` always starts from `context.Background()`, so a run recovered after restart (L143) or stolen by the HA claim sweep gets a fresh trace id with no link to the original. No field stores a run's trace id. | `workflow_handler.go:143, 214-216` | +| **`BuildTrace` consumes 5 of 18 journal entry types** — `step:started`, `step:completed`, `step:failed`, `step:retrying`, `state:changed`. Sleeps, awakeables, sub-workflow relationships and foreach structure never appear in the persisted trace. | `internal/workflow/trace_builder.go:22-68` vs the 18 constants in `internal/workflow/journal.go:15-49` | +| **Doubled function id in the concurrency/rate-limit key.** `workflow_func.go:101` builds `fmt.Sprintf("%s/%s", PackageID, FunctionID)` but `FunctionID` is already the full id, yielding `fuse/pkg/logic/fuse/pkg/logic/sum`. It is used only as the bucket key (L103, L109); the Prometheus label (L114, L132, L167) correctly uses the plain `FunctionID`. Mitigating: those metadata fields are never populated, so the branches are unreachable today. **No backlog task owns it** — file it. | `workflow_func.go:101-109` | +| **`LOG_LEVEL` is not read.** `ParamsConfig.LogLevel` has no `env` tag; the level comes only from the cobra `--loglevel`/`-l` flag. `LOG_FORMAT` **is** tagged (default `json`) and `--log-format` overrides it when non-empty. | `internal/app/config/config.go`; `internal/app/cli/root.go` | +| **Log-format ordering hazard.** `internal/logging/app_logger.go` holds a package-global mutable `logFormat`, written only by `NewAppLogger`. `ErgoLogger()` and the `NewFxLogger()` closure both read it through `newLogger()`, and fx builds its logger early — so fx lifecycle lines can be JSON while the rest of the process is console. Do not add a second format switch. | `internal/logging/app_logger.go`, `fx_logger.go`, `ergo_logger.go` | +| **ergo panic-level logs are downgraded on purpose** (`gen.LogLevelPanic` → zerolog Error) because `zerolog.Panic()` re-panics and can crash the node adapter. Do not "restore" it. | `internal/logging/ergo_logger.go` | +| **`.agents/rules/03-actor-patterns.mdc` uses a method that does not exist**: `a.Log().Warn(...)`. The real ergo API is `Warning` — the codebase uses `a.Log().Warning(` and never `a.Log().Warn(`. | grep the tree | +| **There is no `GET /v1/workflows/{workflowID}/status` route**, and no run-state endpoint returning audit data or a `logs` field. The real route is `/v1/workflows/{workflowID}` and its response DTO carries only workflow id and status. | `internal/actors/mux_worker.go`; `internal/dtos/workflow.go` | + +Inside actors log via `a.Log()` / `h.Log()`. Code that runs outside an actor — node functions, +goroutines, transports — uses the global zerolog `log` (see `transport/internal.go`, `ai/chat.go`, +`http/request.go`). + +## Commands (confirmed to exist) + +```bash +make lint && make build && make test # the mandated gate, in this order +make swagger # REQUIRED on a fresh clone before build/test +go test -v ./internal/tracing/... # 6 provider tests — every one builds from noopConfig() (OTel DISABLED) +go test -v ./internal/metrics/... +go test -v ./internal/logging/... +go test -v -run TestBuildTrace ./internal/workflow/ +make build && ./bin/fuse server -l debug --log-format console +curl -s http://localhost:9090/metrics | grep -E '^fuse_|^ergo_' +curl -s http://localhost:9090/v1/workflows/<workflowID>/trace | jq . +curl -s 'http://localhost:9090/v1/schemas/<schemaID>/traces?limit=10' | jq . +make e2e-local # builds fuse-app:test, compose --profile e2e, -tags=e2e, down -v +E2E_API_URL=http://localhost:9091 go test -tags=e2e ./tests/e2e -run TestE2E_GET_workflow_trace -v -count=1 +``` + +$ARGUMENTS diff --git a/.agents/skills/persistence-and-migrations/SKILL.md b/.agents/skills/persistence-and-migrations/SKILL.md new file mode 100644 index 0000000..c6b4286 --- /dev/null +++ b/.agents/skills/persistence-and-migrations/SKILL.md @@ -0,0 +1,279 @@ +--- +name: persistence-and-migrations +description: Explains how FUSE persists state — the repository interface/memory/postgres triad and its parity contract, the real table shapes, the exact idiom for adding a migration and how `fuse migrate` applies it, the object-store ref convention, and the driver combinations that are silently unrecoverable. Use when adding or changing a repository, writing a migration, touching Config.Validate or the object store, or working F-03, A-05 or A-06. +--- + +# persistence-and-migrations + +Backlog ids below are from the repo-root backlog (`BACKLOG_V2.md`). *If it has been renamed to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog".* + +## When to use + +- Adding a column, a table, or a repository method. +- Anything that reads or writes a payload through the object store. +- F-03 (config validation for driver combinations), A-05 (pin runs to a schema version), + A-06 (stop the shared-pointer live edit). + +## Ground truth + +| File | Authoritative for | +| --- | --- | +| `internal/repositories/<name>.go` | The interface + its `Err…NotFound` sentinels | +| `internal/repositories/<name>_memory.go` | The memory driver | +| `internal/repositories/postgres/<name>.go` | The Postgres driver + its object-store key builder | +| `internal/repositories/postgres/migrations/` | 11 embedded migrations, `000001`…`000011` | +| `internal/repositories/postgres/db.go` | `RunMigrations` (L19) + `convertDSNForMigrate` (L50) | +| `internal/app/di/repos.go` | Driver selection for all 9 repositories | +| `internal/app/di/objectstore.go` | Object-store driver switch | +| `internal/app/config/config.go` | Every env var; `Validate()` at L175 | +| `tests/functional/` | The driver-parity contract harness | +| `docs/adr/0003-in-memory-repositories-by-default.md`, `docs/adr/0019-object-store-payload-externalization.md` | Why there are two of everything, and the ref convention | + +⚠️ `.agents/rules/05-repositories.mdc` is **stale**: it still states FUSE has in-memory +implementations only ("New types can satisfy the same interfaces later if a durable store is +introduced"). The Postgres driver has existed since migration `000001`. Treat `tests/functional/` as +the real convention. + +## The triad and the parity contract + +Every repository is **interface + `Memory*` + `postgres.*`**, selected in `internal/app/di/repos.go` +by one guard repeated for each provider: + +```go +if p.Config.Database.Driver == config.DBDriverPostgres && p.Pool != nil { /* postgres */ } +// else memory +``` + +`config.DBDriverPostgres` (`config.go:13`) is the **only** driver constant; `"memory"`, +`"filesystem"` and `"s3"` are bare string literals at their use sites. + +Parity is enforced by shared contract functions, not duplicated tests: + +```go +// tests/functional/graph_repository_test.go (UNTAGGED — runs in `make test`) +func contractTestGraphRepository(t *testing.T, newRepo func() repositories.GraphRepository, reset func()) +func TestMemoryGraphRepository_Contract(t *testing.T) { contractTestGraphRepository(t, repositories.NewMemoryGraphRepository, func(){}) } + +// tests/functional/postgres_test.go (//go:build functional — runs in `make test-functional`) +func TestPostgresGraphRepository_Contract(t *testing.T) { /* same body, TRUNCATE reset closure */ } +``` + +Eight contracts exist today: Graph, Journal, Workflow (+ SubWorkflowRefs), Package, Environment, +Credential, Claim (Postgres-only), Awakeable. `postgres_test.go` creates schema `fuse_functional`, +runs the migrations into it via a `search_path` DSN (`withSearchPath`, L35) and truncates between +tests. **Behaviour you add belongs in the shared body**, so both drivers are held to it. + +⚠️ **Embedding the interface in the struct defeats compile-time completeness checks.** +`MemoryGraphRepository`, `MemoryWorkflowRepository`, `postgres.WorkflowRepository`, +`postgres.GraphRepository` and `postgres.PackageRepository` all embed their interface (the pattern +`.agents/rules/05-repositories.mdc` prescribes). Add a method to an interface, implement it in one +driver only, and everything still compiles — the other driver nil-panics at runtime. Only +`postgres/environment.go` and `postgres/credential.go` use `var _ repositories.X = (*T)(nil)`. +**Implement both drivers in the same commit and add the case to the shared contract test.** + +## Real table shapes (read from the migration SQL) + +`workflows` — `000001` L34-44, plus `snapshot_ref` (`000003`) and `environment` (`000009`): + +```sql +id BIGSERIAL PRIMARY KEY, workflow_id VARCHAR(36) NOT NULL UNIQUE, +schema_id VARCHAR(128) NOT NULL, state workflow_state NOT NULL DEFAULT 'untriggered', +output_ref VARCHAR(512), claimed_by VARCHAR(128), claimed_at TIMESTAMPTZ, +created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +-- + snapshot_ref VARCHAR(512) (000003) +-- + environment VARCHAR(128) NOT NULL DEFAULT 'default' (000009, indexed) +``` + +**There is no version column.** That is exactly A-05. + +`journal_entries` — `000001` L69-90: `workflow_id VARCHAR(36)`, `sequence BIGINT`, +`entry_type journal_entry_type`, `thread_id SMALLINT`, `function_node_id VARCHAR(128)`, +`exec_id VARCHAR(64)`, `state workflow_state`, `parent_threads SMALLINT[]`, +`input_ref` / `result_ref` / `data_ref VARCHAR(512)`, FK to `workflows(workflow_id)`, and +**`CREATE UNIQUE INDEX idx_journal_wf_seq ON journal_entries (workflow_id, sequence)`**. + +`graph_schema_versions` — `000007`: `(schema_id, version, definition_ref, created_by, comment, +is_active)` with `uq_schema_version UNIQUE (schema_id, version)`, plus +`graph_schemas.active_version INTEGER NOT NULL DEFAULT 1` and a back-fill `INSERT … SELECT`. +A-05's retrieval half is already built on top of this (see below). + +`awakeables` — `000001` L139-154: already carries `deadline_at TIMESTAMPTZ`, which is what the +expiry sweeper **proposed** by C-06 would read. No sweeper exists today. + +Column-type conventions: `VARCHAR(36)` workflow/awakeable ids, `VARCHAR(64)` exec ids, `VARCHAR(128)` +schema/node/package ids and environment names, `VARCHAR(512)` **every** object-store ref, +`SMALLINT` thread ids, `SMALLINT[]` parent threads, `BIGINT` sequences and ns durations, `BYTEA` ciphertext. + +## Adding a migration — copy the newest existing one + +Files are `internal/repositories/postgres/migrations/NNNNNN_snake_title.{up,down}.sql`, 6-digit +zero-padded, strictly sequential, **both files always present** even when the down is a comment +(`000004`). They are picked up by `//go:embed migrations/*.sql` (`db.go:15`) — there is no list to update. +**The current head is `000011`, so the next number is `000012`.** + +The canonical ADD COLUMN idiom is `000009_add_workflow_environment`, verbatim: + +```sql +-- ADR-0031 Phase 3: environment becomes a per-execution scoping dimension. +-- ... 2-4 lines saying WHY, and why the default backfills in-flight rows. +ALTER TABLE workflows ADD COLUMN environment VARCHAR(128) NOT NULL DEFAULT 'default'; +CREATE INDEX idx_workflows_environment ON workflows (environment); +``` + +```sql +DROP INDEX IF EXISTS idx_workflows_environment; -- index first +ALTER TABLE workflows DROP COLUMN IF EXISTS environment; -- IF EXISTS on both +``` + +`NOT NULL` + `DEFAULT` is mandatory when the table can already hold in-flight rows. CREATE TABLE +follows `000001`/`000010`: `id BIGSERIAL PRIMARY KEY`, business key `VARCHAR(n) NOT NULL UNIQUE`, +`created_at`/`updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()`, named constraints `fk_<short>` / +`uq_<short>`, indexes `idx_<table>_<discriminator>`. (`000011_create_credentials` deliberately breaks +the BIGSERIAL rule — its `id` is the business `VARCHAR(128)` PK. Not the norm.) + +**How it is applied.** `postgres.RunMigrations(dsn)` (`db.go:19`) opens the embedded FS via `iofs`, +runs golang-migrate with the pgx/v5 driver, tolerates `migrate.ErrNoChange`. It is reached from two +places: eagerly at fx **Provide** time inside `providePgxPool` (`internal/app/di/database.go`, after +`pool.Ping`, with a comment that it must precede package registration), and from +`fuse migrate` (`internal/app/cli/migrate.go`), which starts neither fx, nor ergo, nor HTTP. + +⚠️ Migrations run **on every node** in the compose `ha` profile (all three run the server). The `e2e` +profile deliberately uses a dedicated `fuse-migrate` init container and gates the nodes on +`service_completed_successfully`. Prefer the e2e shape when concurrent DDL matters. + +⚠️ Migrations target whatever `search_path` the DSN carries (`fuse_functional` for functional tests, +`fuse_e2e` for the e2e stack). A migration that hard-codes a schema-qualified name breaks both. + +⚠️ **Enum migrations are effectively irreversible.** `000004` adds a `journal_entry_type` value and its +`.down.sql` is a comment explaining that PostgreSQL cannot remove enum values. A new journal entry +type needs its own `ALTER TYPE … ADD VALUE` migration. + +## Memory-driver pointer-sharing hazards (A-06) + +**Memory repos hand out live pointers; Postgres repos hand out fresh objects.** + +| Site | Behaviour | +| --- | --- | +| `internal/repositories/graph_memory.go:31-39` `FindByID` | `return graph, nil` — the exact `*workflow.Graph` that `Save` stored (L48) and that `workflow_handler.go:168` hands to `internalworkflow.New` (L177) | +| `internal/repositories/postgres/graph.go:37-67` `FindByID` | rebuilds a `Graph` from `schemas/{id}/definition.json` every call — structurally immune | +| `workflow_memory.go` `Get` | returns the stored `*Workflow` | +| `awakeable_memory.go` `FindByID` | returns the stored `*Awakeable` | + +So `GraphService.updateVersioned` calling `graph.UpdateSchema(schema)` +(`internal/services/graph_service.go:260` → `internal/workflow/graph.go:122-133`, which replaces +`g.schema` and re-runs `compute()` in place) rewrites a **running** run's topology under the memory +driver. That is F-01.3 / **A-06**. + +The fix idiom already exists further down the same file, in `FindByIDAndVersion` +(`graph_memory.go:78-99`): lines 89-90 do +`schema := sv.Schema.Clone(); workflow.NewGraph(&schema)`. For `FindByID`, `graph.Schema()` +(`internal/workflow/graph.go:148-150`) already returns a `Clone`. Expect +`IsNodesMetadataPopulated()` to go false afterwards so `GraphService.FindByID` repopulates — that is +the same cost Postgres already pays, not a regression. + +⚠️ **`GraphSchema.Clone()` is not a total deep copy** (`internal/workflow/graph_schema.go:47-77`): it +deep-copies `Nodes`, `Edges`, `Metadata`, `Tags`, and value-copies `Concurrency`/`TriggerConfig`, but +assigns `Timeout: f.Timeout` — the `*GraphTimeoutConfig` pointer is **shared**. It is correct for +topology, not for everything. + +## Object store + +`pkg/objectstore/store.go` — `Put/Get/Delete/Exists`, keyed by a hierarchical string path; `Delete` is +documented idempotent; `Get` returns `ErrObjectNotFound`. Three drivers, chosen by +`OBJECT_STORE_DRIVER` in `internal/app/di/objectstore.go`: `"filesystem"`, `"s3"`, and a `default` +branch that falls through to **memory with no validation** — a typo yields the memory store silently. + +**The ref convention (ADR-0019): the relational row never holds a payload.** The blob goes to the +object store under a hierarchical key and the row stores that key in a `*_ref VARCHAR(512)` column. +Key builders are unexported functions at the top of the owning Postgres file — add a new one there +rather than inlining a `Sprintf`: + +| Builder | File | Key | +| --- | --- | --- | +| `journalObjectKey` | `postgres/journal.go:28` | `workflows/%s/journal/%d/%s` (`input.json`, `result.json`, `data.json`) | +| `workflowOutputKey` | `postgres/workflow.go:28` | `workflows/%s/output.json` | +| `graphObjectKey` / `graphVersionObjectKey` | `postgres/graph.go:32` / `:16` | `schemas/%s/definition.json`, `schemas/%s/v%d/definition.json` | +| `awakeableResultKey` | `postgres/awakeable.go:29` | `awakeables/%s/result.json` | +| `traceObjectKey` | `postgres/trace.go:29` | `workflows/%s/trace/%s/%s` | +| `packageObjectKey` | `postgres/package.go:31` | (see file) | + +⚠️ Payload PUTs happen **inside** the DB transaction and are not rolled back — `postgres/journal.go` +`Append` writes up to three objects per entry before the INSERT. ADR-0019 accepts the orphans and +notes there is no GC. Journal keys are **sequence-addressed, not content-addressed**; A-09's +content-addressing idea changes this key contract, so dedup must happen at `Put` time. + +## Driver combinations that are silently unrecoverable (F-03) + +`Config.Validate()` (`config.go:175-182`) today checks **one** thing: etcd endpoints when +`CLUSTER_ENABLED=true` and `CLUSTER_DISCOVERY_MODE=etcd`. Everything below is unguarded. + +| Combination | What happens | Evidence | +| --- | --- | --- | +| `DB_DRIVER=postgres` + `OBJECT_STORE_DRIVER=memory` (**the default object store**) | Durable journal rows whose `input_ref`/`result_ref`/`data_ref` point into a process-local map. Every payload fetch fails on the next boot; nothing is recoverable. This is F-03's headline case. | `postgres/journal.go` Append PUTs / LoadAll GETs; `di/repos.go` injects `p.Store` into the Postgres repos with no cross-check | +| `DB_DRIVER=postgres` + empty `DB_POSTGRES_DSN` | `providePgxPool` logs `Warn` and returns a **nil pool**; every provider takes the `Pool != nil` false branch and hands back a `Memory*` repo. The engine boots, looks healthy, persists nothing. | `internal/app/di/database.go` (`if cfg.Database.PostgresDSN == ""` → warn + empty result) | +| `HA_ENABLED=true` + `DB_DRIVER != postgres` | `MemoryClaimRepository.ClaimWorkflow` returns `true` unconditionally — zero mutual exclusion. | `internal/repositories/claim_memory.go:20-21` | + +⚠️ **Coverage hole you must design around: `Config.Validate` is only reached by `fuse server`.** Its +sole caller is `internal/app/fuse.go:47` inside `app.NewApp`, provided by `FuseAppModule`. +`fuse migrate` (`cli/migrate.go`) and `fuse seed examples` (`cli/seed.go`, whose module list omits +`FuseAppModule`) never call it. Either call `Validate` in those `RunE` bodies too, or hoist the check +into a provider inside `DatabaseModule`/`ObjectStoreModule`, which both CLIs do include. + +Style to follow (`config.go:175-182`): a flat sequence of `if <bad combination> { return +fmt.Errorf("<VAR> is required when <VAR>=<v> and <VAR>=<v>") }`, returning on the first failure, +naming **environment variables**, not Go field names. The table test belongs in +`internal/app/config/config_test.go` next to `TestConfig_Validate_EtcdRequiresEndpoints` (L39). + +## A-05 landing map (PROPOSED — none of this exists yet) + +What **exists today**: `FindByIDAndVersion` is fully implemented on `GraphService` and on both +repositories (`graph_memory.go:78-99`, `postgres/graph.go:180-211`) and has **no handler caller** — +A-05 gets the retrieval half for free. What does **not** exist: any record of the version a run +started on. `postgres/workflow.go:Get` → `loadGraph(ctx, schemaID)` (L241) resolves +`SELECT definition_ref FROM graph_schemas WHERE schema_id=$1` — the **active** definition — so a run +recovered after an edit replays against current topology. + +Mechanical shape: new `000012_add_workflow_schema_version.{up,down}.sql` following the `000009` idiom; +add the column to the `Get` SELECT list and to the `Save` INSERT, **excluded from the +`ON CONFLICT … DO UPDATE SET`** exactly like `environment` and with the same style of comment +(`postgres/workflow.go` Save carries that comment today); resolve `loadGraph` by `(schema_id, version)` +against `graph_schema_versions.definition_ref`; mirror in `workflow_memory.go`; extend +`contractTestWorkflowRepository`. + +## Traps + +| Trap | Evidence | +| --- | --- | +| **Postgres enforces `UNIQUE (workflow_id, sequence)` on the journal; memory does not.** A replay bug that re-appends an existing sequence (A-07) errors loudly under Postgres and passes silently under memory. Never conclude "it works" from `make test` alone for journal changes. | `idx_journal_wf_seq` (`000001`); `journal_memory.go` Append is a plain slice append | +| **Awakeable `Resolve` semantics diverge.** Postgres only resolves rows still `pending` (`WHERE … AND status='pending'`, `RowsAffected()==0` → `ErrAwakeableNotFound`); memory overwrites unconditionally. Double-resolve is rejected under one driver and accepted under the other. | `postgres/awakeable.go:133-158` vs `awakeable_memory.go:55-65` | +| **`MemoryWorkflowRepository.FindExecutions` ignores `From`/`To` and returns zero timestamps**, while Postgres filters on `created_at`. | `workflow_memory.go` vs `postgres/workflow.go` FindExecutions | +| **`OBJECT_STORE_KEY_PREFIX` is dead config.** `ObjectStoreConfig.KeyPrefix` (`config.go:128`) is declared and referenced nowhere else — no driver, no key builder applies it. No multi-tenant key isolation comes from it. | grep `KeyPrefix` | +| **`LOG_LEVEL` is dead config.** `ParamsConfig.LogLevel` has **no** `env` tag; the level comes only from the cobra `-l/--loglevel` flag. `LOG_FORMAT` **is** tagged and works. | `config.go:77-82` | +| **`convertDSNForMigrate` never converts `postgresql://`.** `db.go:54` compares `dsn[:14]` against the 13-character literal `"postgresql://"`, which can never be equal, so such a DSN reaches golang-migrate unconverted and fails there rather than at config parse time. `postgres://` (L51) is fine. **No backlog task owns this** — file it with evidence, don't fix it in passing. | `internal/repositories/postgres/db.go:50-57` | +| **`setupTestPool`'s TRUNCATE list is incomplete** — it omits `environments`, `credentials`, `secrets`, `idempotency_keys`, `execution_traces`, `execution_trace_steps` and `graph_schema_versions`. A new table needs an explicit decision about that list and the per-test `reset` closures. | `tests/functional/postgres_test.go:48-92` | +| **The S3 driver has no contract test.** `pkg/objectstore/store_test.go:testObjectStore` runs for memory and filesystem only; `s3_test.go` covers endpoint normalisation alone. S3 behaviour is exercised only by the e2e stack against rustfs. | `pkg/objectstore/*_test.go` | +| **`make test-functional` passes vacuously without a DSN** — `testDSN` calls `t.Skip` when `DB_POSTGRES_DSN` is unset, so every Postgres contract test silently skips. | `tests/functional/postgres_test.go:25-32` | +| **`pkg/store` is not persistence.** It is the in-memory KV backing `Workflow.aggregatedOutput` and dot-notation input mapping. Do not confuse it with `pkg/objectstore`. | `pkg/store/kv.go` | + +## Commands (confirmed to exist) + +```bash +make lint && make build && make test # the mandated gate, in this order +make swagger # REQUIRED on a fresh clone before build/test +make infra-up # docker compose --profile infra (PG 17, rustfs S3, etcd) +DB_POSTGRES_DSN='postgres://fuse:fuse@localhost:5432/fuse?sslmode=disable' make test-functional +go test -count=1 -run 'TestMemory.*_Contract' ./tests/functional/... +go test -tags=functional -count=1 -run TestPostgresGraphRepository_Contract ./tests/functional/... +go test -count=1 ./internal/app/config/... # where the F-03 table test belongs +go test -count=1 ./pkg/objectstore/... +make migrate # go build then ./bin/fuse migrate (needs DB_POSTGRES_DSN) +make seed # ./bin/fuse seed examples -l debug +make e2e-local # builds fuse-app:test, compose --profile e2e, -tags=e2e, down -v +psql "$DB_POSTGRES_DSN" -c '\d workflows' # confirm the live column set before writing a migration +``` + +A schema change additionally needs `make test-functional` against a live PG — that is what actually +exercises the migration, and CI runs it as a separate job after the lint/build/test job. + +$ARGUMENTS diff --git a/.agents/skills/remote-node-protocol/SKILL.md b/.agents/skills/remote-node-protocol/SKILL.md new file mode 100644 index 0000000..2dd430e --- /dev/null +++ b/.agents/skills/remote-node-protocol/SKILL.md @@ -0,0 +1,340 @@ +--- +name: remote-node-protocol +description: Inventories every primitive the FUSE extension protocol will be built on — the transport enum nobody reads, the async result constructor, the unvalidated callback route, ExecutionInfo's real field list, the metadata-only registration dead end, retry/timeout/coercion machinery a remote call inherits — and separates that shipped reality from the eight decisions B-01 must settle. Use when working B-01, B-02, B-05, B-08 or C-10, when designing anything that crosses the engine↔worker boundary, or before claiming that FUSE "supports HTTP transport". +--- + +# remote-node-protocol + +The extension protocol is FUSE's product pillar and it is **half-built and undocumented**. This pack +is the inventory an SDK-facing spec must be written on top of. + +Backlog ids are from the repo-root backlog (`BACKLOG_V2.md`). *If it has been renamed to +`BACKLOG.md`, the canonical file is the one titled "FUSE — product shape and backlog."* + +> **The two halves are two different kinds of claim.** **§ TODAY** is shipped behaviour, every line +> traceable to a `file:line`. **§ TO BE DECIDED** is net-new design owned by B-01 — nothing there +> exists. Quoting a § TO BE DECIDED clause as engine behaviour is the worst failure mode in this area. + +## When to use + +- **B-01** writing the spec (route to the `fuse-protocol-spec-author` agent), **B-02** implementing + the remote transport, **B-05** the conformance suite, **B-08** worker lifecycle, **C-10** the + per-node idempotency key, and **A-03** (replay of pending remote steps), which the spec must state. +- Any time someone reads `pkg/transport.HTTP` and concludes HTTP transport exists. + +**Out of scope here:** the in-process function contract (`function-package-authoring`), replay and +journal mechanics (`durable-execution-internals`), spans (`observability-tracing`). **B-06/B-07 — +the TypeScript and PHP SDKs — live outside this repo entirely.** + +## Ground truth + +| File | Authoritative for | +| --- | --- | +| `pkg/transport/type.go` | The entire transport enum — 11 lines, no logic | +| `internal/packages/transport/function.go` | `FunctionTransport`, the seam a remote transport implements | +| `internal/packages/transport/internal.go` | The only implementation; where `Finish` is bound | +| `internal/packages/loaded_package.go` / `registry.go` | Registration mapping, the executable/metadata-only branch, the non-downgrade rule | +| `pkg/workflow/execution_info.go` | Every field a function receives — five | +| `pkg/workflow/fn_result.go` / `fn_output.go` | The result constructors and the one status enum | +| `internal/handlers/async_function_result.go` | The external callback route | +| `internal/messaging/execute_function.go` | The closest existing analogue to an invocation envelope | +| `internal/typeschema/parse.go` | The complete set of coercible declared types | +| `internal/idempotency/store.go` | The idempotency contract (trigger-level only, ADR-0017) | +| `docs/adr/0024`, `0022`, `0023`, `0017`, `0027` | Registry contract, retry/error, timeouts, idempotency scope, the `ExecutionInfo` prohibition | + +--- + +## § TODAY — every primitive that exists + +### The transport layer + +| Primitive | Location | Reality | +| --- | --- | --- | +| `transport.Type` enum | `pkg/transport/type.go:5` | Wire-level, public, JSON-visible | +| `HTTP Type = "http"` | `pkg/transport/type.go:9` | **Never read by production code.** Its only appearances in the tree are 8 inert fixture values in `tests/functional/package_repository_test.go` | +| `gRPC Type = "grpc"` | `pkg/transport/type.go:10` | **Lowercase `g` → unexported.** Unreachable outside `pkg/transport`. Dead code, not a stub | +| `Internal` | `internal/packages/transport/type.go:6` | A *different* package. The only transport that executes | +| `FunctionTransport` | `internal/packages/transport/function.go:10-18` | `Execute(actor.Handle, *ExecutionInfo)` + `ExecuteSync(*ExecutionInfo)` | +| `InternalFunctionTransport` | `internal/packages/transport/internal.go:35` | The only implementation | + +No production code branches on `transport.Type` at all — the only two comparisons in the tree are +`loaded_package.go:112` and `agent_tools.go:101`, both against `Internal`. + +### Registration — and the trap that makes `Transport` metadata meaningless + +`MapToRegistryPackage` (`internal/packages/loaded_package.go:63`) builds the registry-side +`FunctionMetadata` with **`Transport: transport.Internal` hardcoded at L70**, whatever the incoming +wire metadata declared. It then branches at **L112**: + +```go +if function.Metadata.Transport == transport.Internal && function.Function != nil { + functions[functionID] = NewLoadedInternalFunction(functionID, metadata, function.Function) +} else { + functions[functionID] = NewLoadedFunction(functionID, metadata) // Transport == nil +} +``` + +⚠️ **Consequence, confirmed by reading:** `packages.FunctionMetadata.Transport` is **always** +`"internal"` in the registry. The only working discriminator between an executable and a +metadata-only entry is `LoadedFunction.Transport == nil` (`loaded_function.go:26-30`). Any B-02/B-03 +predicate written against `Metadata.Transport` will silently match everything. This also makes the +second clause of `isExposableTool` (`agent_tools.go:101`) vacuously true today. + +Dispatch of a metadata-only function fails at `loaded_package.go:41` / `:56` with the verbatim text: + +``` +function %s has no transport (package likely loaded from persistence without its code-backed function) +``` + +`Registry.Register` (`registry.go:47-65`) will **discard** any incoming function whose `Transport` is +nil in favour of a previously registered executable entry, so a remote registration that does not +produce a non-nil `Transport` is silently thrown away. The rule is load-bearing and regression tested +(`internal/packages/loaded_package_test.go:83`). + +### What a function receives, and what it returns + +`pkg/workflow/execution_info.go:14-23` — **five fields**: `WorkflowID`, `ExecID`, `Environment`, +`Input`, `Finish`. No `context.Context`, no nodeId, no threadId, no attempt, no deadline, no +idempotency key. `Finish` is nil until a transport binds it (`internal/packages/transport/internal.go:47`). + +`ExecID` (`pkg/workflow/exec_id.go:10`) is a UUIDv8 with the thread id packed into bytes 6–7; +`ExecID.Thread()` (L23) recovers it via `hex.DecodeString` whose error is **discarded**, then indexes +`raw[6]`/`raw[7]`. + +| Result constructor | `Async` | Meaning | +| --- | --- | --- | +| `NewFunctionResult(status, data)` (`fn_result.go:10`) | false | general sync result | +| `NewFunctionResultSuccess()` / `…SuccessWith(data)` | false | success | +| `NewFunctionResultError(err)` (`:36`) | false | returns `(result, nil)` — a **nil Go error** | +| `NewFunctionResultAsync()` (`:44`) | true | "I will call `Finish` later"; output is `success` with nil data | + +The status enum is three values (`fn_output.go:6-10`), of which the engine branches on two: +`FunctionSuccess` and `FunctionError`. + +### The two async completion paths + +**In-process (works).** Return `NewFunctionResultAsync()` and later call `execInfo.Finish(output)`. +`Finish` is bound by `InternalFunctionTransport.Execute` (`internal.go:47`) and delivers via +`sendAsyncFunctionResult` (`internal.go:18`) — `gen.Node.Send` to +`gen.Atom(actornames.WorkflowHandlerName(wfID))`, **by registered name, not PID** (`internal.go:12-17`). + +**External callback (exists, unvalidated).** + +``` +POST /v1/workflows/{workflowID}/execs/{execID} +{"result": {"status": "success", "data": {}}} +``` + +Route: `internal/actors/mux_worker.go:79` — POST only, 10s timeout, `PoolSize: 3` (the route entry +spans `:76-86`). +Handler: `internal/handlers/async_function_result.go:53-85`. +Body DTO: `dtos.AsyncFunctionRequest{Result workflow.FunctionOutput}` (`internal/dtos/workflow.go:29`) +— one field, **no validate tag**, no attempt, no signature, no idempotency field. + +⚠️ **Defect, do not present as the contract.** The handler reads two path params, binds JSON, and +`Send`s. It does **not** check that the workflow exists, that the exec is pending, that the exec +belongs to that workflow, that the execID is well-formed, or that a result was not already +submitted; it returns `200 {"code":"OK"}` whenever the actor `Send` succeeds. The e2e test documents +this (`tests/e2e/apis_e2e_test.go:201-217`: "workflow handler may not exist → 500 is acceptable for +smoke"). **B-02 owns fixing it.** The validation template to copy is +`internal/handlers/resolve_awakeable.go:66-73`, which 404s on missing and 400s on non-pending. + +⚠️ **A duplicate callback advances the graph twice.** `Workflow.SetResultFor` +(`internal/workflow/workflow.go:422`) overwrites `entry.Result`, re-sets `aggregatedOutput`, and +appends a **second** `step:completed`; `handleMsgAsyncFunctionResult` then calls `Workflow.Next` +again. Its only guard is `a.isTerminalState()` (`workflow_handler.go:319`). + +⚠️ **An async dispatch is journalled `step:completed` immediately.** +`handleMsgFunctionResult` calls `SetResultFor` at **`workflow_handler.go:275`, before** the +`if fnResultMsg.Result.Async` check at **`:277`**. So the moment a node returns +`NewFunctionResultAsync()`, the journal already carries a `step:completed` for it with the +placeholder empty output, and `aggregatedOutput[nodeID]` is set to nil. This inverts the naive +assumption behind A-03: `findPendingThreads` (`workflow.go:267`, `step:started` without completion) +will **not** see an in-flight remote call as pending. Settle it against the F-02 harness before +choosing replay semantics. + +### What a remote node inherits for free + +| Mechanism | Location | Notes | +| --- | --- | --- | +| Per-node retry | `internal/workflow/retry.go` (ADR-0022) | `RetryPolicy{MaxAttempts (0–100), Backoff{fixed\|exponential\|linear, InitialInterval, MaxInterval, Multiplier}}`; default 3 / exponential / 1s→30s / ×2 | +| Retry dispatch | `workflow.go:841 HandleNodeFailure` | Reuses the **same `ExecID`**; attempt count lives only in the in-memory `RetryTracker` (`retry_tracker.go`) — **A-08** owns journalling it | +| Error routing | `EdgeSchema.OnError` (`edge_schema.go`), `findErrorEdges` (`workflow.go:899`) | Fires only **after** retries are exhausted, and only `errorEdges[0]` is followed | +| Per-node timeout | `internal/workflow/timeout.go` `TimeoutConfig.Execution` (ADR-0023) | `startExecutionTimeout` (`workflow_handler.go:622`) → `ExecutionTimer` (`execution_timer.go`), in-memory `SendAfter`; zero = no timeout. Lost on restart — **C-06** | +| Timeout failure shape | `workflow_handler.go:551,564` | Synthesises `FunctionError` with `Data{"error": "execution timeout exceeded"}` and reuses the ADR-0022 failure path | +| Input coercion | `internal/typeschema/parse.go:20 ParseValue` | Exactly `string`, `int`, `float64`, `bool`, `[]byte`, `map[string]any`, plus `[]T` of those; anything else → `unsupported type: %s` | +| Schema binding | `internal/services/graph_service.go:295 populateNodeMetadata` | Splits `node.Function` at the **last** `/`, `Registry.Get(pkgID)`, `GetFunctionMetadata(fullID)` | + +⚠️ **Coercion failures are swallowed.** Every failure branch in `inputMapping` (`workflow.go:652`) +and `applyFlowMapping` (`:727`) is a log line (`log.Error`, in one case `log.Warn`) followed by +`continue`, `break` or `return` — none returns an error to the caller. A type mismatch produces a +**missing parameter**, not a failed node. A declared capability schema therefore buys type +*coercion*, not type *enforcement*. + +⚠️ Two shipped declarations already use a type `ParseValue` rejects: `headers` +(`internal/packages/functions/http/request.go:66-67`) and `data` +(`internal/packages/functions/system/wait.go:23`) both declare `Type: "map"`, not `"map[string]any"`. + +### Idempotency, as it actually is + +`internal/idempotency/store.go:7-19` — `Check` / `Set` / `Delete` / `CheckAndSet`. Keys are **opaque +strings**; the store knows nothing about execs, nodes or attempts. ADR-0017 states the scope limit +explicitly: dedup is at the **trigger** boundary. + +| Producer | Key shape | Atomicity | +| --- | --- | --- | +| HTTP trigger | client-supplied `idempotencyKey` | ⚠️ non-atomic `Check` (`trigger_workflow.go:79`) then `Set` (`:104`) — two concurrent identical requests can both pass | +| Cron | `cron:<schemaID>:<RFC3339 minute>` (`cron_scheduler.go:86`) | atomic `CheckAndSet` | +| Event | `evt:<schemaID>:<type>:<source>:<sha256(json(data))[:16]>` (`event_trigger.go:120-124`) | atomic `CheckAndSet` | + +Nothing derives a key from `(workflowID, execID, attempt)`. That is **C-10**. + +### Registration over the API is lossy in both directions + +`internal/dtos/package.go` has **no `Edges` field on `InputMetadataDTO` or `OutputMetadataDTO`** +(grep confirms zero hits for `Edges` in that file). So `FromPackagedFunctionDTO` +(`internal/dtos/package.go:150`) yields a function with **zero declared input or output edges** — an +API-registered capability cannot declare conditional or error routing at all — and a `GET → PUT` +round-trip loses data. `FunctionMetadataDTO.Transport` even carries the swagger example `"sync"` +(`internal/dtos/package.go:36`), which is not a valid `transport.Type`, and +`FromPackagedFunctionDTO` casts that string straight through (`:178`) with no validation. +**No backlog task names the dropped-edges defect.** B-01 decision 7 (capability declaration) is its +natural home, but that is this pack's inference and not something the backlog says — file it with +evidence rather than fixing it inside another task. This paragraph is the single source for the +registration-loss inventory; `function-package-authoring` points here rather than restating it. + +⚠️ **Per-function concurrency and rate limiting are unreachable from any registration path.** +`packages.FunctionMetadata` declares `Concurrency` and `RateLimit` +(`internal/packages/function_metadata.go:14-15`) and `WorkflowFunc.HandleMessage` reads both +(`workflow_func.go:102-109`), but the wire type `workflow.FunctionMetadata` +(`pkg/workflow/metadata.go:6-10`) has only `{Transport, Input, Output}`, the DTO has neither field, +and `MapToRegistryPackage` never sets them — so both are always nil for every registered package. +This is not "DTO loss"; it is a field pair with a consumer and no producer. + +⚠️ `PUT /v1/packages/{packageID:.+}` **ignores its path parameter**: `register_package.go:61` reads +it and uses it only in log/error messages; the saved id comes from the JSON body. There is **no +DELETE route for packages** — that route declares `GET, PUT` only (`mux_worker.go:120-121`; other +resources do declare `DELETE`) — though `postgres/package.go:162` implements `Delete`. + +### Auth: greenfield + +`docs/API.md` says verbatim "**Authentication** — Not required today". There is no middleware in +`internal/actors/mux_server.go`; `Handler.SendJSON` sets `Access-Control-Allow-Origin: *` +(`internal/handlers/handler.go:59`). The only credential machinery that exists is `pkg/secrets`: +`SecretValue` redacts in every String/JSON sink and only `Reveal()` yields plaintext; +`CredentialSecretName(id, field)` resolves to `cred/<id>/<field>` (`pkg/secrets/credential.go:15`); +`workflow.Credential` carries field **names** only. + +--- + +## § TO BE DECIDED — the eight B-01 decisions and what constrains each + +**None of this is built.** Each row names the engine constraint that makes the choice cheap or +load-bearing. + +| # | Decision | Constraint from existing code | Cost to reverse | +| --- | --- | --- | --- | +| 1 | **Push or pull** | Nothing exists for pull: the only worker-facing inbound surface is the async-result callback route, and no route declared in `mux_worker.go` is a lease/poll/stream endpoint. Push matches "register your existing API as a node" | **Highest in the document.** Backlog recommends push first, pull later for locked-down networks | +| 2 | **Invocation envelope** | Closest analogue `messaging.ExecuteFunctionMessage` (`execute_function.go:12-20`) has 7 fields and lacks attempt, deadline, callback URL and idempotency key; `traceparent` does exist but one level up, on `messaging.Message.TraceCarrier` (`message.go:44`), not on the envelope. Only 4 of the 7 survive into `ExecutionInfo`. `nodeId` exists only as `entry.FunctionNodeID` on audit-log/journal entries; `threadId` is on the message and derivable via `ExecID.Thread()` | Medium — additive fields are cheap, renames are not | +| 3 | **Ack semantics** | One bit: `FunctionResult.Async`. Async parks the run at `workflow_handler.go:277` — but see the `SetResultFor`-before-the-check trap above | Medium | +| 4 | **Completion taxonomy** | **No representation today.** One status plus a free-form `Data["error"]`; timeout, rate-limit rejection and "no transport" all synthesise the identical shape. Needs a **new wire field**, not a convention | High — retry safety depends on it | +| 5 | **Delivery guarantee (A-03)** | Retries reuse the same `ExecID`; manual `RetryNode` (`workflow.go:451`) mints a new one; attempt count is in-memory only. Requires A-08 before a key can include attempt | High — SDK dedup logic is downstream | +| 6 | **Auth, both directions** | 100% greenfield. Secret plumbing exists (`pkg/secrets`), transport-level identity does not | High — crosses a trust boundary | +| 7 | **Capability declaration** | Metadata-only packages **already** bind into graphs (`populateNodeMetadata`), so schema load is not the gap. Type set is capped by `ParseValue`'s six. Edge metadata is dropped by the REST DTO | Medium — schema shape is public | +| 8 | **Versioning** | No representation: `workflow.Package{ID, Functions, Tags}`. The only versioning is graph-schema versioning (`internal/workflow/versioned_schema.go`), pinned per run by A-05 | High — every SDK negotiates on it | + +--- + +## B-02 implementation sketch (after B-01 is published) + +1. **New transport** implementing `internal/packages/transport/function.go:FunctionTransport`, in a + new file beside `internal.go`. Do **not** add a dispatch switch elsewhere — + `LoadedPackage.ExecuteFunction` already delegates blindly to `function.Transport`. +2. **`NewLoadedRemoteFunction`**, a peer of `NewLoadedInternalFunction` (`loaded_function.go:17`), + producing a **non-nil** `Transport` — otherwise `Registry.Register`'s non-downgrade rule discards it. +3. **`MapToRegistryPackage`**: stop hardcoding `Transport: transport.Internal` at **L70**, and give + the `else` at **L118** a remote arm. Both edits are required; either alone leaves the registry + unable to tell a remote capability from an internal one. +4. **Release the pool worker.** `WorkflowFunc.HandleMessage` calls `pkg.ExecuteFunction` + synchronously (`workflow_func.go:136`) and the pool is `PoolSize: 3` **per workflow instance** + (`workflow_func_pool.go:40`). A blocking remote call holds a third of that run's dispatch + capacity for its whole duration. The transport must return `NewFunctionResultAsync()` and resume + on callback — this is exactly **A-10**. +5. **Validate the callback.** Add existence / pending / duplicate / format checks in + `internal/handlers/async_function_result.go`, modelled on `resolve_awakeable.go:66-73`, and + remember `SetResultFor` is not idempotent. +6. **Inject `traceparent`.** Blocked on **A-04**: `workflow_func.go:96` is literally `_ = nodeCtx` + and `ExecutionInfo` has no `context.Context`. ADR-0027 forbids re-adding a per-execution runtime + *handle* as an `ExecutionInfo` field; plain data fields are a different question the A-04 ADR must + settle. +7. **Persistence already indexes it.** `postgres/package.go:151` writes + `package_functions(package_id, function_id, transport)`. +8. **Fix or scope the metadata gap.** Edges are dropped by the REST DTO in both directions; per-function + `Concurrency`/`RateLimit` have a consumer (`workflow_func.go:102-109`) and no producer at all. A + remote capability that cannot declare an error edge cannot participate in the ADR-0022 failure + model, and one that cannot declare a concurrency cap will hammer the customer's endpoint. + +## B-05 — the conformance-suite scenarios and what each pins + +| Scenario | Pins | Existing anchor | +| --- | --- | --- | +| Sync result | `FunctionResult{Async:false}` reaches the handler | `workflow_func.go:172` | +| Async callback | `POST /v1/workflows/{id}/execs/{execID}` resumes the run | `workflow_handler.go:313` | +| Business error | routes to the **first** `OnError` edge, after retries | `workflow.go:899` | +| Infrastructure error | **indistinguishable today** — needs the B-01 taxonomy field | — | +| Retry with a stable key | backoff schedule + key stability across attempts | `retry.go DelayFor`; note the reused `ExecID` | +| Trace propagation | `traceparent` in and out | blocked on **A-04** | +| Deadline handling | what the worker does when its deadline passes | `timeout.go`; ADR-0023's caveat: the remote side keeps running | +| Schema validation | declared types coerce; mismatches must not silently vanish | `typeschema/parse.go`; `applyFlowMapping` swallow | +| Restart behaviour | per the **A-03** decision | `findPendingThreads` + the async `step:completed` trap | + +Harness precedent: `tests/functional/package_repository_test.go:13 contractTestPackageRepository` is +the contract-test shape (one body, both drivers); `tests/e2e/` is the black-box HTTP shape +(`//go:build e2e`, polls `E2E_API_URL`, never boots a server). + +## B-08 — worker lifecycle: the open questions + +Nothing exists. Registration is a plain upsert (`package_service.go:80` — `Validate` → repo `Save` → +`Registry.Register`): no health check, no deregister, no drain, no TTL, no heartbeat, and no package +DELETE route. `FindAll`/`FindByID` with `Load: true` backfill the registry **only** `if !packageRegistry.Has(id)`, +so an already-registered capability is never refreshed by a GET — updates must go through `Save`. +"Worker unreachable" has no error class and would surface as a generic `FunctionError`, retried by +the node's `RetryPolicy`. Multiple workers behind one capability has no addressing model +(`workflow.Package` carries `{ID, Functions, Tags}`). Versioned rollout interacts with +`versioned_schema.go` and **A-05**. + +## C-10 — the derived idempotency key + +`ExecutionInfo` already carries `WorkflowID` and `ExecID`, so two of three components are present. +The third is the problem: automatic retries **reuse** the same `ExecID`, `RetryTracker` is in-memory +and never journalled as a number, and manual `RetryNode` mints a fresh `ExecID`. The attempt must be +threaded through `workflowactions.RetryFunctionAction.Attempt` (which already exists, set in +`HandleNodeFailure`) into `messaging.ExecuteFunctionMessage` and then into `ExecutionInfo` — that is +**A-08**'s dependency. Follow the shape of the existing deterministic keys (`cron:…`, `evt:…`). + +## Commands (confirmed to exist) + +```bash +make swagger # REQUIRED on a fresh clone: docs/docs.go is gitignored but blank-imported (mux_server.go:19) +make lint && make build && make test # the mandated gate, in this order +go test -v ./internal/packages/... # registry, mapping, non-downgrade rule +go test -v -run TestRegister_DataOnlyCopy_DoesNotDowngradeExecutableFunc ./internal/packages/ +go test -v ./internal/typeschema/ ./internal/idempotency/ +go test -v -run TestHandleNodeFailure ./internal/workflow/ +DB_POSTGRES_DSN=... make test-functional # -tags=functional ./tests/functional/... ; skips vacuously without the DSN +make e2e-local # docker build + compose --profile e2e + go test -tags=e2e ./tests/e2e -timeout 15m +make run # ./bin/fuse server on :9090 +# NB: GET /v1/packages serves the *persisted* workflow.Package (packages.go:52 FindAll → ToPackageDTO), +# NOT the registry-side metadata. It echoes whatever transport string was stored, so it does NOT show +# the registry's hardcoded "internal" — only a Go test or a debugger can observe that. +curl -s localhost:9090/v1/packages | jq '.items[].functions[].metadata.transport' +curl -X POST "localhost:9090/v1/workflows/$WF/execs/$EXEC" \ + -H 'content-type: application/json' -d '{"result":{"status":"success","data":{}}}' # the unvalidated callback +``` + +⚠️ There is no `make examples-ci` target and no `scripts/run-example-workflows.sh`; the real entry +points are `make seed` and `./bin/fuse seed examples --ci`. And there is no +`GET /v1/workflows/{workflowID}/status` route — it is `GET /v1/workflows/{workflowID}`, returning +`{workflowId, status}` only. + +$ARGUMENTS diff --git a/.claude/agents b/.claude/agents new file mode 120000 index 0000000..4c8a5fc --- /dev/null +++ b/.claude/agents @@ -0,0 +1 @@ +../.agents/agents \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index f500b63..e25bcf0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,17 +9,45 @@ Project guidance lives in a tool-agnostic home and is shared with every tool via - **`.agents/rules/`** — coding rules (Go conventions, actors, repositories, handlers, testing, DI, concurrency, …). Indexed in [`.agents/rules/README.mdc`](.agents/rules/README.mdc). -- **`.agents/skills/`** — reusable skill packs (architecture patterns) plus ADR-authoring - skills (`write-adr`, `spec-to-adr`). +- **`.agents/skills/`** — reusable skill packs (architecture patterns), ADR-authoring skills + (`write-adr`, `spec-to-adr`), and the engine knowledge packs listed below. +- **`.agents/agents/`** — subagent definitions for the backlog delivery pipeline (below). - **`.agents/commands/`** — GitHub Spec Kit commands (`speckit.*`). - **`docs/adr/`** — Architecture Decision Records ([index](docs/adr/README.md)); see [ADR-0009](docs/adr/0009-portable-ai-agent-guidance.md) for why guidance lives in `.agents/`. - **`.specify/memory/constitution.md`** — project constitution (governing principles). -Tool wiring: `.cursor/{rules,skills,commands}` and `.claude/{rules,skills,commands}` are +Tool wiring: `.cursor/{rules,skills,commands}` and `.claude/{rules,skills,commands,agents}` are symlinks into `.agents/`; `CLAUDE.md` imports this file via `@AGENTS.md`. Edit content under `.agents/` (the single source of truth), never the symlinks. +## Delivering backlog work + +The work queue is [`BACKLOG_V2.md`](BACKLOG_V2.md) ("FUSE — product shape and backlog"). +`BACKLOG.md` is **superseded** and its task ids were renumbered wholesale — never cite them. + +Start with the **`backlog-task`** skill (`/backlog-task F-02`). It runs one task through +verify → research → plan → implement → review → gates, delegating to these subagents: + +| Stage | Subagent | +| ----- | -------- | +| Reproduce the claim with an executed failing test (never fixes it) | `fuse-premise-verifier` | +| Read-only cross-subsystem research, CONFIRMED vs INFERRED per claim | `fuse-engine-researcher` | +| Verified premise → plan (tests, driver parity, HA, migration, ADR/spec call) | `fuse-implementation-planner` | +| Write the remote node protocol spec — Tier B, before any Go | `fuse-protocol-spec-author` | +| Test-first implementation under `.agents/rules/` + the quality gates | `fuse-go-implementer` | +| Real-process SIGKILL/restart durability harness (F-02) | `fuse-e2e-harness-engineer` | +| Adversarial review: replay, actors, races, parity, scope, contract | `fuse-code-reviewer` | + +Engine knowledge packs, loaded by tier: `durable-execution-internals` and `crash-resume-testing` +(F/A), `persistence-and-migrations` (F-03, A-05, A-06), `observability-tracing` (A-04, C-04), +`function-package-authoring` (in-process nodes), `remote-node-protocol` (Tier B — the pillar), +`capability-registry-and-agents` (B-03/B-04, Tier C). + +Two rules the backlog treats as non-negotiable: **a premise that does not reproduce closes its +task** (that is a success, not a failure), and **once the B-01 protocol spec is published it is a +contract with strangers** — treat it with more care than engine internals. + ## Learned User Preferences - Prefer dev tooling to be installable from the Makefile so `make test` and `make lint` work on fresh machines without manual binary setup. @@ -31,11 +59,11 @@ symlinks into `.agents/`; `CLAUDE.md` imports this file via `@AGENTS.md`. Edit c - golangci-lint must be built with a Go version at least as new as `go.mod`; use `make install-lint` (`go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.5.0`). Prebuilt install scripts can embed an older Go and fail with a version-mismatch error when loading `.golangci.yml`. - `make test` runs `install-gotestsum` first (`go install gotest.tools/gotestsum@latest` into `GOPATH/bin`). - With ergo v3.2+, sending to sibling processes during actor `Init` is supported; workflow init runs inline in `WorkflowHandler.Init` instead of an `ActorInit` / `actor:init` self-message. Async completion from goroutines outside `HandleMessage` (e.g. timer callbacks) should deliver messages via `Node().Send`, not `Process.Send`, when the worker can be in Sleep (ergo restricts `Process.Send` to Init/Running/Terminated). -- `make examples-ci` and `scripts/run-example-workflows.sh` use `CI=true` to skip `github-request-example` and workflows that reference `fuse/pkg/logic/timer` where the default server path does not complete async timer flows in CI. Multiple files under `examples/workflows/` reuse the same schema `id` (e.g. `small-test`); upserting them in order means the last file wins for that id. -- `GET /v1/workflows/{workflowID}/status` returns workflow instance state and audit data from the in-memory workflow repository; an optional `logs` field appears when the server log level is debug. +- Example workflows are seeded with `./bin/fuse seed examples --ci`, where `--ci` skips `github-request-example` and any workflow referencing `fuse/pkg/logic/timer`, because the default server path does not complete async timer flows in CI. (A `make examples-ci` target and `scripts/run-example-workflows.sh` were removed; `--ci` is the surviving equivalent.) Multiple files under `examples/workflows/` reuse the same schema `id` (e.g. `small-test`); upserting them in order means the last file wins for that id. +- Run state is read with `GET /v1/workflows/{workflowID}` (`internal/actors/mux_worker.go:220`); the sibling routes are `/cancel`, `/snapshot`, `/retry-node`, `/retry` and `/trace`. There is **no** `/status` route. The bare route returns `{workflowId, status}` only — richer mid-flight state is backlog task C-09. - `MemoryPackageRepository` uses a mutex because internal package registration saves packages from concurrent goroutines. - `make dockerfile-lint` runs Hadolint in Docker on `Dockerfile` with `.hadolint.yaml`; it catches consecutive-`RUN` patterns that align with SonarCloud Docker rules (e.g. docker:S7031). `make sonar-local` uses `sonar-project.properties` and Dockerized SonarScanner; set `SONAR_TOKEN` from SonarCloud (analysis-capable token, not a GitHub token). It passes the current git branch and commit; if analysis fails with HTTP 404 on `analysis/analyses`, try `SONAR_REGION=us` for US-hosted SonarCloud orgs. - Default filesystem object store path is `./data/fuse` (`OBJECT_STORE_FS_BASE_PATH`) so local runs on macOS do not try to create `/data` on the read-only root volume; with Docker and the repo mounted at `/app`, the same default resolves under `/app/data/fuse`. Use an explicit absolute path (for example `/data/fuse`) in production when a volume is mounted there. -- `fuse migrate` applies embedded PostgreSQL migrations using `DB_POSTGRES_DSN` without starting the HTTP server or actor runtime; `make migrate` builds then runs `./bin/fuse migrate`. `fuse seed examples` upserts every `examples/workflows/*.json` graph schema through `GraphService` with the same DB and object-store settings as the server, but without HTTP or ergo; use `--ci` to skip `github-request-example.json` and any file whose JSON references `fuse/pkg/logic/timer`, matching `make examples-ci` / `scripts/run-example-workflows.sh`. After CLI changes, rebuild `bin/fuse`—an older binary will not expose new subcommands. +- `fuse migrate` applies embedded PostgreSQL migrations using `DB_POSTGRES_DSN` without starting the HTTP server or actor runtime; `make migrate` builds then runs `./bin/fuse migrate`. `fuse seed examples` upserts every `examples/workflows/*.json` graph schema through `GraphService` with the same DB and object-store settings as the server, but without HTTP or ergo; use `--ci` to skip `github-request-example.json` and any file whose JSON references `fuse/pkg/logic/timer`. After CLI changes, rebuild `bin/fuse`—an older binary will not expose new subcommands. - All Docker infrastructure is in a single `docker-compose.yml` with profiles: `infra` (PG+S3+etcd only), `ha` (3 Fuse nodes built from source), `e2e` (3 nodes from pre-built image with migrate init). Use `make infra-up`, `make ha-up`, or `docker compose --profile e2e up`. - `LOG_FORMAT` env var (default `json`) controls log output format. Set `LOG_FORMAT=console` for human-readable colorized output (local dev). The CLI flag `--log-format` overrides the env var when provided. JSON mode uses RFC3339 timestamps and disables ANSI colors in all loggers (app, ergo, fx). diff --git a/BACKLOG.md b/BACKLOG.md new file mode 100644 index 0000000..9dfae49 --- /dev/null +++ b/BACKLOG.md @@ -0,0 +1,335 @@ +# FUSE core — implementation backlog + +> **SUPERSEDED — do not work from this file.** Replaced by [`BACKLOG_V2.md`](BACKLOG_V2.md) +> ("FUSE — product shape and backlog"). This version was written for a separate TypeScript +> platform sitting on top of the engine; that framing is gone and **every task id was +> renumbered**. Citing an id from this file will send you to the wrong task. Kept only for +> provenance of the Phase 0 findings. + +Derived from the Phase 0 capability report (`core/ @ 31c43e1`). Every task cites the evidence it came from. + +**Read this first.** These tasks are written against a *report about* the code, not the code. Several findings are explicitly marked as inference in the report. **Task F-01 exists to verify before anything is built.** If a task's premise fails verification, close it and say so — do not implement around a bug that does not exist. + +Tiers are dispatch order, not priority theatre: + +- **F — Foundation.** Verification and the safety net. Nothing else starts until these pass. +- **A — Blocks Phase 3.** Without these, the ThinkAssist slice cannot run on FUSE at all. +- **B — Blocks the pilot.** The slice runs without them; a paying customer does not. +- **C — Post-pilot.** Real, scheduled, not now. + +Sizes: **S** ≈ under a day, **M** ≈ a few days, **L** ≈ a week or more with design risk. + +--- + +## Tier F — Foundation + +### F-01 · Verify the three high-leverage findings +**Size S · no dependencies** + +Three findings carry disproportionate weight and two of them are read from control flow rather than executed. Confirm each with a failing test before writing any fix. + +1. `internal/actors/workflow_func.go` — the node span context is discarded (`_ = nodeCtx`), so function code cannot create child spans or inject `traceparent` outbound. +2. Restart with a pending `system/subworkflow` step spawns a **second child workflow** — `findPendingThreads` matches `step:started` without completion, `replayPendingThread` re-issues the action, interception fires again. +3. `internal/repositories/graph_memory.go:FindByID` returns the shared `m.graphs[id]` pointer that live `Workflow`s hold, so `graph_service.go:updateVersioned` → `graph.UpdateSchema` rewrites a running run's topology in place. + +**Done when:** each is a checked-in test that fails on `main`, or a written note that the finding did not reproduce and why. + +### F-02 · Crash-resume e2e harness +**Size M · depends on nothing · blocks everything in A** + +The report is unambiguous: no test kills the process mid-run and asserts correct resume. `Workflow.Resume()` is called from one place (`workflow_handler.go:156`) and has no unit test. Everything Stellwerk sells rests on this path. + +Build a harness that runs FUSE as a real process against real Postgres + fs object store, kills it with SIGKILL at a controllable point, restarts, and asserts final state. Not in-process — the existing `tests/e2e/workflow_resilience_suite_e2e_test.go` already covers in-process failure and that is not the same thing. + +Scenarios, each its own test: +- kill after a completed step, mid-thread → resumes, no re-execution of the completed node +- kill with a pending async node → asserts exactly-once *or* documents at-least-once explicitly +- kill with a pending `system/subworkflow` → **exactly one child exists after restart** (this is F-01.2's regression test) +- kill with a pending `system/sleep` → remaining duration is honoured, not restarted +- kill with a pending awakeable → the token issued before the crash still resolves +- kill a multi-thread run between thread completions → all branches resume (covers the `buildResumeAction` inference) +- restart three times on the same run → journal length and rendered trace are stable + +**Done when:** the harness runs in CI, and each scenario is either green or has a linked task in this backlog. Several will be red at first — that is the point. Do not fix them here. + +### F-03 · Config validation for driver combinations +**Size S** + +`DB_DRIVER=postgres` with the default `OBJECT_STORE_DRIVER=memory` produces durable journal rows in Postgres whose `input_ref`/`result_ref`/`data_ref` point into a store that dies with the process. `LoadAll` then fails on every payload fetch and no run is recoverable. Silent, and it looks correct until the first restart. + +Extend `Config.Validate` (`internal/app/config/config.go`) to reject persistent-DB + ephemeral-object-store, and any other mismatched pair. Fail at boot with a message that names both variables. + +**Done when:** boot fails fast on the mismatch; test covers each combination. + +--- + +## Tier A — Blocks Phase 3 + +### A-01 · Input payload on workflow trigger +**Size S · high value, low risk** + +`dtos.TriggerWorkflowRequest` is `{schemaID, idempotencyKey, environment}` — no payload. The plumbing is built and then dropped: `messaging.TriggerWorkflowMessage` has `Input`, `NewTriggerWorkflowWithInputMessage` exists, webhook/cron/event triggers populate it — but `workflow_sup.go:spawnWorkflowActor` takes three args, `WorkflowHandlerInitArgs` has three fields, and `workflow.go:Trigger` hardcodes `Args: map[string]any{}`. + +Thread `Input` through: DTO → handler → `TriggerWorkflowMessage` → `spawnWorkflowActor` → `WorkflowHandlerInitArgs` → `Trigger(Args: input)`. Make it reachable from `SourceFlow` mappings under a reserved key (`trigger.*` or similar) so schemas can map trigger input into the first node. + +**Done when:** `POST /v1/workflows` with a body reaches the first node's inputs; e2e asserts it; the same key works after a restart (it is journaled with the run). + +**Why it blocks:** without it, every run starts by calling back into TypeScript to fetch its own input. One extra HTTP hop, one extra failure mode, on every single turn. + +### A-02 · Generic async HTTP node +**Size M · the single most important new capability** + +The transport layer is in-process only — `pkg/transport/type.go` declares HTTP and gRPC constants with no implementation, and `loaded_package.go:MapToRegistryPackage` registers non-internal functions metadata-only ("function %s has no transport"). The synchronous `fuse/pkg/http/request` blocks a pool worker for the call duration, defaults to a 10s timeout, and three concurrent long calls exhaust a run's 3-worker pool. + +Meanwhile the async primitive already exists and no node uses it: `NewFunctionResultAsync()` plus `POST /v1/workflows/{workflowID}/execs/{execID}`. + +Build `fuse/pkg/http/request_async`: POST to a configured URL, return `NewFunctionResultAsync()`, resume on callback. Requirements: +- the outbound request carries `workflowId`, `execId`, `nodeId`, `threadId`, and the callback URL in the body — **the execID is the callback token**, which sidesteps the awakeable discoverability problem (B-01) entirely +- `traceparent` injected into outbound headers (depends on A-05) +- per-node timeout with a durable deadline (depends on B-02) — for Phase 3 an in-memory timer is acceptable if the limitation is written down +- does not hold a pool worker while waiting +- callback validates that the exec is genuinely pending and rejects duplicates + +**Done when:** a workflow with this node survives a 90-second TypeScript call, and survives a restart while the call is in flight (behaviour under restart is asserted, whichever behaviour you choose — see A-03). + +### A-03 · Decide and enforce replay semantics for pending async nodes +**Size M · depends on A-02, F-02** + +`findPendingThreads` matches `step:started` without completion — exactly the shape of an in-flight async node — and re-issues the action. So after a restart the TypeScript side gets called a second time with a **new execID**, and the original callback is orphaned. + +Two options; pick one, write it in an ADR, enforce it in code: + +- **(a) At-least-once, TS-side idempotent.** Cheap. The outbound call carries a stable `(workflowId, nodeId, attempt)` key; TypeScript dedupes; the old execID's callback is rejected as stale. Requires B-05 to give callers a stable key. +- **(b) Replay-aware.** Extend `replayJournalEntries` to recognise pending async steps and re-arm the callback with the *original* execID instead of re-dispatching. Correct, and the same machinery A-04 needs. + +Given A-04 lands anyway, (b) is probably the honest answer — but (a) unblocks Phase 3 faster. Decide explicitly rather than by omission. + +**Done when:** the corresponding F-02 scenario is green and the ADR states which guarantee FUSE offers. + +### A-04 · Replay the intercepted system functions +**Size M · depends on F-02** + +`replayJournalEntries` handles five of eighteen journal entry types: `thread:created`, `step:started`, `step:completed`, `thread:finished`, `state:changed`. `JournalAwakeableCreated`, `JournalSleepStarted` and `JournalSubWorkflowStarted` each have one write site and **zero read sites**. The rows are written and never read back. + +Consequences, in descending severity: +- **subworkflow** — restart spawns a duplicate child. A real duplicate side effect. +- **awakeable** — restart re-mints the token via `uuid.New()`; the ID you handed the client is orphaned pending forever. +- **sleep** — restart restarts the full duration from zero. + +Add replay handlers for all three. Minimum for Phase 3 is **subworkflow** (duplicate side effect) — awakeable and sleep can follow in B if the Phase 3 design uses A-02 for gates instead of awakeables. + +**Done when:** the three F-02 scenarios are green. + +### A-05 · Trace context end to end +**Size M** + +The report calls TS → Go → TS three disconnected traces. Three separate breaks: + +1. **Inbound.** `internal/handlers/trigger_workflow.go:HandlePost` never reads request headers; `startRootSpan` calls `tracer.Start(context.Background(), ...)`. The provider has `ExtractCarrier` and it is never called at the HTTP boundary. → extract `traceparent` from the trigger request and parent the root span to it. +2. **Handler → worker.** Already works (`InjectCarrier` on `ExecuteFunction`, `ExtractCarrier` in `workflow_func.go`). Leave alone, add a regression test. +3. **Outbound.** `_ = nodeCtx` in `workflow_func.go:96` discards the span context. `ExecutionInfo` has no `context.Context` field, so function code cannot create children or inject `traceparent`. → add the context to `ExecutionInfo` and use it in A-02. + +Apply to every entry point that starts a run: HTTP trigger, webhook, event, cron (cron has no inbound parent — give it a documented root). + +**Done when:** one trace spans TypeScript → FUSE root → node → outbound HTTP → TypeScript, verified against a real collector, not a unit test. + +### A-06 · Pin runs to a schema version +**Size M** + +`STELLWERK.md` §4 rule 2 requires published versions to be immutable and runs pinned. Today they are not: the `workflows` table has `schema_id` and **no version column**, and `postgres/workflow.go:Get → loadGraph(ctx, schemaID)` reads the *active* definition. A run recovered after an edit replays against whatever topology is current. Rollback is worse — it writes a new active version, so in-flight runs get their topology swapped silently, unrecorded in their own journal. + +The retrieval half already exists: `FindByIDAndVersion` is implemented on the service and both repositories. The run simply never records where it started. + +- migration: add `schema_version` to `workflows` +- record it at trigger time, journal it in the run's first entry +- `loadGraph` resolves by `(schemaID, version)` +- rollback and version-activate must not touch in-flight runs + +**Done when:** a run started on v3 completes on v3 after v4 is published and activated mid-run; test asserts it across a restart. + +### A-07 · Stop the shared-pointer live edit +**Size S · depends on F-01.3** + +`graph_memory.go:FindByID` returns `m.graphs[id]` — the same pointer live `Workflow`s hold — so a `PUT /v1/schemas/{id}` rewrites a running run's nodes, edges and thread assignments mid-execution. The postgres driver rebuilds a fresh `Graph` per call and is safe in-memory. + +Return a deep copy from the memory driver, matching postgres semantics. Assert both drivers behave identically. + +**Note:** A-06 fixes the *durable* half of this (resume against the right version). A-07 fixes the *live* half. Both are needed; neither substitutes for the other. + +**Done when:** editing a schema mid-run leaves the running instance's topology untouched under both drivers. + +### A-08 · Fix duplicate journal append on replay +**Size S** + +`replayJournalEntries` calls `SetResultFor`, and `SetResultFor` appends a journal entry. `Journal.LoadFrom` sets `lastPersisted = seq`, so replay-generated duplicates get fresh sequence numbers and are persisted by `persistWorkflowState()` at the end of `Init`. The journal grows by a full copy of its completed steps on every restart, and `trace_builder.go:BuildTrace` renders each step N+1 times after N restarts. + +Suppress journaling during replay — a replay flag on the workflow, or a `SetResultForReplay` that skips the append. The former is less surface area. + +**Done when:** journal length and rendered trace are byte-identical after three restarts (the F-02 stability scenario). + +**Why it blocks:** it compounds. Each restart makes the next one heavier, and it corrupts the trace — which is the thing you are selling. + +--- + +## Tier B — Blocks the pilot + +### B-01 · Expose awakeable IDs +**Size S** + +The token is minted inside the engine and written only into a journal entry's `Data` map. `GET /v1/workflows/{id}` returns `{workflowId, status}`; `SnapshotTimelineEvent` deliberately drops `Data`; snapshot and trace are persisted only at terminal state. `AwakeableRepository.FindPending(workflowID)` exists with no route in `mux_worker.go`. + +Add `GET /v1/workflows/{id}/awakeables` (pending only) and include the ID in whatever B-03 exposes. + +**Note:** if Phase 3 uses A-02 for gates, awakeables are not on the critical path — but they are a documented FUSE feature that is currently unusable by any external caller. As the maintainer of a public OSS engine, that is a bug regardless of what Stellwerk needs. + +### B-02 · Durable timers and an expiry sweeper +**Size M** + +Sleeps, awakeable timeouts and workflow timeouts are all per-actor `SendAfter`. They vanish on restart and there is no sweeper — the only periodic actors are `workflow_claim_actor.go` and `cron_scheduler.go`. A 72-hour human gate that outlives a deploy has no deadline at all. + +The `Awakeable` row already carries `DeadlineAt`. Add a periodic sweeper that scans expired pending awakeables and expired sleeps, and re-arms in-memory timers for live ones on boot. HA-aware: sweeping must respect claims so two nodes do not fire the same timeout. + +**Done when:** a gate with a 60s timeout, opened before a restart that takes 90s, fires its timeout after boot. + +### B-03 · Mid-flight run read model +**Size M** + +`GET /v1/workflows/{id}` returns `{workflowId, status}`. Snapshot and trace are persisted only in `sendWorkflowCompleted` — absent or stale for the entire life of a run. No SSE, no WebSocket; `internal/events/memory_bus.go` is in-process and unreachable from TypeScript. + +Stellwerk's own design sidesteps this by keeping the read model in TypeScript, fed by push. But FUSE cannot be operated or debugged without it, and every other consumer of the engine has the same hole. + +Minimum: `GET /v1/workflows/{id}/state` returning current thread positions, completed steps, pending steps and open gates, built from the journal. Streaming is a separate, later task — do not conflate them. + +### B-04 · ForEach state survives restart +**Size M** + +Iteration bookkeeping lives in `WorkflowHandler.forEachStates` / `iterThreadToForEach` — plain Go maps, never reconstructed from the journal — so a ForEach in flight does not survive a restart. The journal already writes `foreach:started`, `foreach:iteration:started`, `foreach:iteration:completed`, `foreach:completed` and never reads them: the same class of bug as A-04. + +Also fix, or document loudly: `spawnForEachBatch` sends `RunFunctionAction` straight to the function pool, **bypassing system-function interception**. So `system/subworkflow`, `system/sleep` and `system/wait` inside a ForEach body hit the no-op placeholder, return success instantly, and silently do nothing. That is a silent-wrong-answer bug — worse than a crash. If the fix is large, ship schema validation that **rejects** those functions inside a ForEach body until it is fixed. + +### B-05 · Per-node idempotency convention +**Size M** + +`internal/idempotency/store.go` is trigger-level dedup only (`trigger_workflow.go`, `cron_scheduler.go`, `event_trigger.go`). Nothing derives a key from `(runId, execId, attempt)`, and the engine's own resume path re-executes pending steps — so at-least-once side effects are the documented behaviour with no dedup layer beneath it. + +`ExecutionInfo` already carries `WorkflowID` and `ExecID` into every function, so the raw material is there. Add a derived, stable idempotency key to `ExecutionInfo`, make the built-in HTTP nodes send it as a header, and document the contract for custom functions. **Stable across replay** is the whole point — see the A-03 decision. + +### B-06 · Distinguish retry attempts +**Size S** + +Automatic retries reuse the same `ExecID` (`HandleNodeFailure` → `RetryFunctionAction{RunFunctionAction{FunctionExecID: execID}}`). Attempts are separable only via `step:retrying` entries and the in-memory `RetryTracker`, which is not rebuilt on replay. Manual retry does it correctly — `RetryNode` mints a fresh ExecID and records `previousExecId`. + +Add an attempt counter to the journal entry and rebuild `RetryTracker` on replay, so "this node failed twice then succeeded" is answerable from durable state. Cost attribution and the trace view both need it. + +### B-07 · Multi-thread resume correctness +**Size M · depends on F-02 · premise is inference** + +`buildResumeAction` walks `lastCompletedThreadIDs` and returns the first non-noop action, so a multi-thread run that crashed between thread completions may lose its other branches. The report flags this as read from control flow with no test either way. + +Verify with the F-02 multi-thread scenario first. If it reproduces, resume must fan out to every pending thread, not the first. + +### B-08 · Claim failure must fail closed +**Size S** + +`claimForThisNode` returns `true` on a claim-store error and runs the workflow anyway (there is a "running anyway" log line). Under Postgres pressure that is a split-brain path — two nodes executing the same run, with side effects. + +Fail closed: on claim-store error, do not claim, log, retry on the next sweep. Also revisit `ClaimWorkflows(nodeID, 10)` every 5s — recovering 50 runs after a node loss takes ~25 seconds. + +### B-09 · Payload write amplification +**Size M** + +Every journal entry carrying a payload is one object-store PUT, with no size threshold, no inlining, no compression, no dedup. A 200 KB conversation transcript is re-PUT on every `step:started` whose input includes it and every `step:completed` whose result includes it, plus a third time into `output.json` on every `Save`. At 50 concurrent runs × ~12 steps that is ~600 PUTs and ~120 MB per round of turns. `LoadAll` then re-fetches the entire payload graph on every resume. + +Two changes, independently valuable: +- **inline small payloads** in the PG row below a threshold, skipping the object store entirely +- **content-address large payloads** by hash, so an unchanged transcript is stored once and referenced N times + +The second one alone removes most of the amplification for conversational workloads, which is exactly Stellwerk's shape. + +### B-10 · Configurable pool size +**Size S** + +`WorkflowFuncPool` is `PoolSize: 3` per workflow instance (`workflow_func_pool.go:Init`). Any blocking node caps a run at three concurrent branches, and one 90-second call holds a third of that run's capacity. Make it configurable globally and per-schema. A-02 reduces the pressure; it does not remove it. + +### B-11 · Guard sub-workflow recursion +**Size S** + +Children spawn through the same `WorkflowSupervisorName` route as any run — no depth counter, no ancestry check, no cycle detection. A self-referencing schema is an unbounded fork bomb. Since Stellwerk models sub-agents as sub-workflows, an agent that can pick its own sub-agent can trigger this from *data*, not from a bad schema. + +Add a depth counter in `SubWorkflowRef`, a configurable max, and ancestry cycle detection at spawn. + +### B-12 · Pass input into sub-workflows +**Size S · depends on A-01** + +`handleSubWorkflowAction` accepts and journals the `input` parameter and then never passes it to the child. Once A-01 exists, wire it: the child's `Trigger` args come from the parent's declared input mapping. + +Also: the completion message carries the child's **entire** per-node output map, unfiltered (`AggregatedOutputSnapshot()`). Add an output selector so a parent can take one path instead of the whole child state — otherwise nested runs balloon the parent's aggregated output, compounding B-09. + +--- + +## Tier C — Post-pilot + +### C-01 · Address node output by execution +**Size L · the only task here that changes the data model** + +`SetResultFor` writes `w.aggregatedOutput.Set(entry.FunctionNodeID, ...)` — the KV is keyed by **node ID, not exec ID**. A node that runs twice overwrites its own previous output, and `SourceFlow` only ever sees the latest execution. The journal keeps every execution; the data plane does not. + +This is the structural mismatch with `STELLWERK.md` §5.2: per-step history is not addressable from inside the graph, so a "reject step 2, keep steps 1, 3, 4" correction cannot be expressed in FUSE's own data plane. + +**Workaround for the pilot:** keep selection history in the TypeScript read model and pass the corrected set back in as fresh input. It works, and it means the *engine* is not the system of record for a correction — which is worth writing down as a known limitation rather than discovering in front of a customer. + +Real fix: key by `(nodeID, execID)` with a `latest` alias for existing schemas, and a mapping syntax for addressing a specific execution. Breaking-ish; needs its own ADR and a migration story. + +### C-02 · Runtime graph mutation +**Size L · only if Phase 1 concludes it is needed** + +Today: no. `NewGraph` computes once from `GraphSchema`; `Workflow` holds that pointer and never re-derives; there is no add-node/add-edge on `Graph`, no such workflow action, no such message. The existing dynamism is conditional routing over declared edges, cycles (genuinely supported by `calculateThreads`), ForEach dynamic threads (width only, over already-declared nodes, linear body), and sub-workflow spawn. + +If Phase 1 lands on the static supervisor loop plus sub-workflows, **close this task**. Do not build it speculatively. Revisit only when a real workflow cannot be expressed. + +### C-03 · Out-of-process function transport +**Size L** + +Make the HTTP and gRPC constants in `pkg/transport/type.go` real, so a node's implementation can be a service in another language and `MapToRegistryPackage` stops registering them metadata-only. A-02 covers Stellwerk's need with a single built-in node; this is the general version, and it is what makes FUSE interesting to people who do not write Go. + +### C-04 · OTel GenAI attributes and cost +**Size M** + +Spans exist (`workflow.execute`, `node.execute`) with `workflow.*` / `node.*` attributes. None of the `STELLWERK.md` §10 set exist — `.agent.id`, `.model`, `.tokens.in/out`, `.cost.usd`, `.gate.decision`, `.tenant.id` — and LLM usage is Prometheus metrics only (`ai/usage.go`), disconnected from the trace. + +Follow OTel GenAI semantic conventions where they exist. **Design note:** most of these attributes belong to Stellwerk's gateway, not to FUSE. Decide the split before implementing, or you will emit them twice from two places and reconcile them forever. + +### C-05 · Graph-level context management +**Size M** + +`ai/agent`'s `maxContextTokens` / `contextStrategy` trims a transcript *inside one node execution*. There is no equivalent at graph level, so a conversation growing across steps grows unbounded in `aggregatedOutput` and in every journal payload. Partly mitigated by B-09; the policy question is separate from the storage question. + +--- + +## Dispatch order + +``` +F-01 ─┬─ F-02 ──┬── A-03, A-04, B-07 (need the harness to define "correct") + └─ A-07 │ +F-03 ────────────┤ + │ +A-01 ─┬─ A-02 ───┘ A-01 also unblocks B-12 + └─ A-05 ── A-02 (outbound traceparent) +A-06 ── independent +A-08 ── independent, do early (compounds every restart) +``` + +**Suggested first sprint:** F-01, F-02, F-03, A-08, A-07. Verification and the safety net, plus the two cheap correctness fixes. Nothing user-visible ships, and every subsequent task becomes measurable. + +**Second sprint:** A-01, A-05, A-02, then the A-03 decision. That is the Phase 3 slice. + +--- + +## Rules for the implementing agent + +- **Verify before fixing.** Every task cites a report, not the code. If the premise does not hold, close the task and report it. +- **One task, one PR.** These are deliberately separable. Do not bundle A-02 with A-05 because they touch adjacent lines. +- **A test that reproduces the bug lands before the fix**, in the same PR, failing in the first commit. +- **Do not fix bugs discovered along the way.** Add them to this backlog with evidence and keep going. Scope creep in an engine rewrite is how three-week projects become three-month ones. +- **Tier C is not a parking lot for hard problems.** If a Tier A task turns out to require C-01, stop and escalate rather than quietly starting it. +- **Public API changes need a note in the changelog and a migration path.** FUSE is Apache-2.0 with users who are not you. diff --git a/BACKLOG_V2.md b/BACKLOG_V2.md new file mode 100644 index 0000000..9c16905 --- /dev/null +++ b/BACKLOG_V2.md @@ -0,0 +1,422 @@ +# FUSE — product shape and backlog + +Replaces the previous backlog, which was written for a separate TypeScript platform sitting on top of the engine. That framing is gone. Everything executes inside FUSE. + +Derived from the Phase 0 capability report (`core/ @ 31c43e1`). Tasks cite the evidence they came from. **The report describes the code; it is not the code.** Task F-01 exists to verify before anything is built. If a premise fails verification, close the task and say so. + +--- + +## 0. Product shape + +**FUSE Core** — Apache-2.0, Go. The engine: graph execution, durable journal, agents, the extension protocol, and the SDKs that speak it. This is the thing people adopt, extend, and talk about. + +**FUSE Enterprise** — commercial layer on top of Core. Console, tenancy, RBAC/SSO, audit, cost accounting, collaboration, packaging. No fork, no divergent engine: Enterprise is a consumer of Core's API like any other. + +Two topologies, and the second is what makes the first credible: + +``` +Most customers: Enterprise ──▶ Core + +Customers who extend: Enterprise ──▶ Core ──▶ their API + (a node, via SDK) +``` + +**The extension protocol is the product pillar, not a feature.** An engine that only runs Go functions is a Go tool. An engine where a Laravel shop registers an existing endpoint as a node in twenty minutes is infrastructure. The async primitives already exist in Core (`NewFunctionResultAsync`, `POST /v1/workflows/{workflowID}/execs/{execID}`) and no node uses them; the transport layer declares HTTP and gRPC and implements neither. The protocol is half-built and undocumented. Finishing and specifying it is Tier B and it is the highest-leverage work in this document. + +**One capability, two projections.** A registered external API must produce *both* a node (callable from a workflow graph) and a tool (callable by an agent) from a single definition. Registering the same endpoint twice, once per consumer, is the obvious mistake and it is very hard to undo later. Design the registry first, project from it second. + +### What this decision costs + +Running agents inside the engine means the conversation lives in the journal, so three things that were previously avoidable are now unavoidable core work: payload write amplification (A-09), pool blocking under long LLM calls (A-10), and per-execution addressing of node output (A-11). LLM latency also moves onto the engine's critical path. That is the price of one system of record, one journal, one trace, one product — the right trade for the positioning, but a real price, and it lands in Tier A rather than being deferred. + +--- + +## Tiers + +- **F — Foundation.** Verification and the safety net. Nothing else starts, and no execution-model decision gets made, until these pass. +- **A — Core correctness.** The engine's durability claims have to be true before anything is sold on them. +- **B — Extension protocol and SDKs.** The product pillar. +- **C — Agents in Core.** What makes it an agent engine rather than a workflow engine. +- **D — Enterprise layer.** Separate repo, separate lifecycle, starts once A is green. + +Sizes: **S** ≈ under a day, **M** ≈ a few days, **L** ≈ a week or more with design risk. + +--- + +## Tier F — Foundation + +### F-01 · Verify the three high-leverage findings +**S · no dependencies** + +1. `internal/actors/workflow_func.go` — node span context discarded (`_ = nodeCtx`), so function code cannot create child spans or inject `traceparent` outbound. **Now critical**, because it is the seam every SDK crosses. +2. Restart with a pending `system/subworkflow` spawns a second child — `findPendingThreads` matches `step:started` without completion, `replayPendingThread` re-issues, interception fires again. +3. `graph_memory.go:FindByID` returns the shared `m.graphs[id]` pointer live `Workflow`s hold, so `updateVersioned → graph.UpdateSchema` rewrites a running run's topology in place. + +**Done when:** each is a checked-in failing test on `main`, or a written note that it did not reproduce and why. + +### F-02 · Crash-resume e2e harness +**M · blocks all of A** + +No test kills the process mid-run and asserts resume. `Workflow.Resume()` is called from one place (`workflow_handler.go:156`) with no unit test. Everything FUSE claims rests on this path. + +Real process, real Postgres, fs object store, SIGKILL at a controllable point, restart, assert. Not in-process — `workflow_resilience_suite_e2e_test.go` covers in-process failure, which is a different thing. + +Scenarios, each its own test: kill after a completed step; kill with a pending async node; kill with a pending `system/subworkflow` (**exactly one child after restart**); kill with a pending `system/sleep`; kill with a pending awakeable (pre-crash token still resolves); kill a multi-thread run between thread completions; three restarts on one run with stable journal length and trace. + +**Done when:** it runs in CI and each scenario is green or has a linked task. Several will be red. That is the point — do not fix them here. + +### F-03 · Config validation for driver combinations +**S** + +`DB_DRIVER=postgres` with default `OBJECT_STORE_DRIVER=memory` yields durable journal rows whose `input_ref`/`result_ref`/`data_ref` point into a store that dies with the process. `LoadAll` then fails on every payload fetch and nothing is recoverable. Silent, and it looks correct until the first restart. + +Extend `Config.Validate` to reject persistent-DB + ephemeral-object-store and any other mismatched pair, naming both variables in the error. + +--- + +## Tier A — Core correctness + +### A-01 · Input payload on workflow trigger +**S** + +`dtos.TriggerWorkflowRequest` is `{schemaID, idempotencyKey, environment}` — no payload. Plumbing is built and dropped: `TriggerWorkflowMessage.Input` exists, `NewTriggerWorkflowWithInputMessage` exists, webhook/cron/event populate it — but `spawnWorkflowActor` takes three args, `WorkflowHandlerInitArgs` has three fields, and `Trigger` hardcodes `Args: map[string]any{}`. + +Thread it end to end and expose it to `SourceFlow` under a reserved key (`trigger.*`). + +**Elevated in this design:** when an agent composes a workflow run, parameterising the trigger *is* the composition mechanism. Without it the agent has no way to pass anything into what it starts. + +### A-02 · Replay the intercepted system functions +**M · depends on F-02** + +`replayJournalEntries` handles five of eighteen journal types. `JournalAwakeableCreated`, `JournalSleepStarted`, `JournalSubWorkflowStarted` each have one write site and **zero read sites**. + +- **subworkflow** — restart spawns a duplicate child. Real duplicate side effect. +- **awakeable** — restart re-mints via `uuid.New()`; the token you handed out is orphaned pending forever. +- **sleep** — restart restarts the full duration. + +All three are blocking now: agents inside the engine will use sub-workflows for composition and awakeables for gates. + +### A-03 · Replay semantics for pending remote steps +**M · depends on A-02, F-02, B-01** + +`findPendingThreads` matches `step:started` without completion — the shape of an in-flight remote call — and re-issues it. After a restart the SDK worker is invoked a second time, with a new execID, and the original callback is orphaned. + +Decide and enforce, in an ADR: **(a)** at-least-once with a stable idempotency key the SDK dedupes on, or **(b)** replay-aware re-arming of the original execID. + +The SDK contract makes (a) far more palatable than it was when the caller was hand-written — dedup can live in the SDK, not in every customer's controller. Whichever you pick, **write it in the protocol spec**, because every SDK author depends on it. + +### A-04 · Trace context end to end +**M** + +Three breaks. Inbound: `trigger_workflow.go:HandlePost` never reads headers, `startRootSpan` uses `context.Background()`; `ExtractCarrier` exists and is never called at the HTTP boundary. Handler→worker: works, add a regression test. Outbound: `_ = nodeCtx` discards the span context and `ExecutionInfo` has no `context.Context`, so function code cannot create children or inject `traceparent`. + +**This is the seam the SDKs cross.** A distributed engine whose selling point is traceability, that breaks the trace at exactly the boundary it invites you to extend across, is unsellable. Add the context to `ExecutionInfo`; B-02 depends on it. + +### A-05 · Pin runs to a schema version +**M** + +`workflows` has `schema_id` and no version column; `postgres/workflow.go:Get → loadGraph(ctx, schemaID)` reads the *active* definition, so a run recovered after an edit replays against current topology. Rollback writes a new active version, silently swapping in-flight runs, unrecorded in their journals. + +`FindByIDAndVersion` already exists on the service and both repositories — the run just never records where it started. Add `schema_version` to `workflows`, record and journal it at trigger, resolve `loadGraph` by `(schemaID, version)`, and leave in-flight runs alone on activate and rollback. + +### A-06 · Stop the shared-pointer live edit +**S · depends on F-01.3** + +Memory driver returns the live pointer; postgres rebuilds per call. Return a deep copy from memory and assert both drivers behave identically. A-05 fixes the durable half of this; A-06 fixes the live half. Both are needed. + +### A-07 · Fix duplicate journal append on replay +**S · do first** + +`replayJournalEntries` calls `SetResultFor`, which appends. `LoadFrom` sets `lastPersisted = seq`, so replay duplicates get fresh sequences and are persisted at the end of `Init`. The journal grows by a full copy of completed steps every restart and `trace_builder.go:BuildTrace` renders each step N+1 times after N restarts. + +Suppress journaling during replay via a replay flag. It compounds, and it corrupts the trace — the thing you sell. + +### A-08 · Distinguish retry attempts +**S** + +Automatic retries reuse the same `ExecID` (`HandleNodeFailure → RetryFunctionAction{RunFunctionAction{FunctionExecID: execID}}`); attempts are separable only via `step:retrying` and the in-memory `RetryTracker`, not rebuilt on replay. Manual retry does it right (`RetryNode` mints a fresh ExecID, records `previousExecId`). + +Add an attempt counter to the journal entry and rebuild `RetryTracker` on replay. Cost attribution per attempt (C-04) and the trace view both need it, and remote nodes make retries far more common. + +### A-09 · Payload write amplification +**M · promoted — agents now put conversations in the journal** + +Every payload-carrying journal entry is one object-store PUT. No threshold, no inlining, no compression, no dedup. A 200 KB transcript is re-PUT on every `step:started` whose input includes it and every `step:completed` whose result includes it, plus into `output.json` on every `Save`. At 50 concurrent runs × ~12 steps: ~600 PUTs, ~120 MB per round. `LoadAll` re-fetches the whole payload graph on every resume, and A-07's duplicates make each restart heavier than the last. + +Two independent changes: **inline small payloads** in the PG row below a threshold; **content-address large payloads** by hash so an unchanged transcript is stored once and referenced N times. The second alone removes most of the amplification for conversational workloads. + +### A-10 · Configurable pool size and non-blocking remote calls +**S–M · promoted** + +`WorkflowFuncPool` is `PoolSize: 3` per instance. Any blocking node caps a run at three concurrent branches; one 90-second LLM or remote call holds a third of that run's capacity. Make it configurable globally and per-schema, and make sure the remote transport (B-02) never holds a worker while awaiting a callback. + +### A-11 · Address node output by execution +**L · promoted from post-pilot — the biggest design risk in this document** + +`SetResultFor` writes `w.aggregatedOutput.Set(entry.FunctionNodeID, ...)` — keyed by **node ID, not exec ID**. A node that runs twice overwrites its own output; `SourceFlow` only ever sees the latest. The journal keeps every execution; the data plane does not. + +With agents inside the engine this stops being theoretical: an agent loop is a cycle, every turn re-executes the same node, and each turn destroys the previous one's output in the data plane. Conversation history, per-turn selections, and any "reject step 2, keep 1, 3 and 4" correction are all unaddressable from inside the graph. + +**Do not start by re-keying the KV.** Two options, and the cheaper one is probably also the better one: + +- **(a) Turn state as a first-class Core concept** — a dedicated store (repository + journal entry types) that agent nodes read and append to, addressed by `(runID, agentNodeID, turn)`, entirely separate from `aggregatedOutput`. Smaller blast radius, no breaking change to existing schemas, and it is a thing you want to exist anyway (C-02). +- **(b) Re-key `aggregatedOutput` by `(nodeID, execID)`** with a `latest` alias for compatibility and mapping syntax for a specific execution. Correct and general; breaking-ish; needs its own ADR and a migration story. + +Decide in an ADR before either. This is the one item that is expensive to reverse. + +--- + +## Tier B — Extension protocol and SDKs + +The pillar. Everything here is net-new and it is what turns Core from a Go library into a platform. + +### B-01 · Specify the remote node protocol +**M · blocks every other B task · do this before writing any Go** + +A written spec, versioned, in the repo, that an SDK author who has never read the engine can implement. Existing raw material: `NewFunctionResultAsync`, `POST /v1/workflows/{workflowID}/execs/{execID}`, `pkg/transport/type.go` constants, `ExecutionInfo{WorkflowID, ExecID, Environment, Input, Finish}`. None of it is documented as a contract, and `PackagedFunction.Function` is `json:"-"`, so API-registered functions are metadata-only today (`MapToRegistryPackage` → "function %s has no transport"). + +The spec must settle: + +1. **Push or pull.** FUSE calls the worker's HTTP endpoint, or the worker polls/streams from FUSE. Push fits NestJS and Laravel idiomatically — they are already HTTP servers — and fits "register your existing API as a node" exactly. Pull works behind firewalls with no inbound reachability, which is how Temporal survives enterprise networks. **Recommendation: push first**, with pull as a later transport for locked-down deployments. Decide explicitly; it is the hardest thing to change later. +2. **Invocation envelope** — `workflowId`, `execId`, `nodeId`, `threadId`, attempt, idempotency key, deadline, environment, input, callback URL, `traceparent`. +3. **Ack semantics** — sync result vs `202 + async callback`. The async path is the interesting one and the reason this exists. +4. **Completion** — success, business error (routes to the error edge), infrastructure error (retryable), and the distinction between them. Getting this taxonomy right is what makes retries safe. +5. **Delivery guarantee** — the A-03 decision, stated plainly, with the idempotency key SDKs dedupe on. +6. **Auth** — how a worker proves identity and how FUSE proves it is FUSE. Both directions, since customers will run this across trust boundaries. +7. **Capability declaration** — input/output schema, so a registered node is typed and `applyFlowMapping`'s coercion (`internal/typeschema/parse.go`) still works. +8. **Versioning** — of the protocol itself and of a registered capability. + +### B-02 · Implement the remote transport in Core +**M · depends on B-01, A-04, A-10** + +Make the HTTP transport real: `MapToRegistryPackage` stops registering non-internal functions metadata-only; a remote node dispatches per B-01, returns `NewFunctionResultAsync()`, releases its pool worker, and resumes on callback. Inject `traceparent`. Validate that the callback's exec is genuinely pending and reject duplicates and stale execIDs. + +This is the generalised, specified version of what the previous backlog called "a generic async HTTP node". Same code, promoted from workaround to product surface. + +### B-03 · Capability registry +**M · depends on B-01** + +One registration produces two projections: a **node** usable in a graph, and a **tool** usable by an agent. Single definition, single schema, single auth config, single version. Registration via API and via schema-as-code. + +Design constraint for the ADR: an agent calling a capability directly and a workflow node calling the same capability must produce the same journal entries, the same span shape, and the same idempotency behaviour. If the two paths diverge, traceability becomes conditional on which one was used — and that is the property Enterprise is sold on. + +### B-04 · MCP ingestion +**M · depends on B-03** + +Import an MCP server's tools as capabilities. Real customer flows already speak MCP — the reference broker assistant is entirely MCP tools returning `{items, systemMessage, userMessage}` — so this is how existing agent estates arrive on FUSE without a rewrite. + +Consider the inverse (**expose** FUSE capabilities as an MCP server) but do not build both at once; ingestion is the one with a customer behind it. + +### B-05 · SDK conformance suite +**M · depends on B-01 · build before the first SDK** + +One executable test suite any SDK must pass: sync result, async callback, business error, infra error, retry with a stable idempotency key, trace propagation, deadline handling, schema validation, restart behaviour per A-03. + +Written once, run against every language. Without it, three SDKs means three subtly different engines and support tickets you cannot reproduce. + +### B-06 · SDK: TypeScript / Node +**M · depends on B-02, B-05** + +Two entry points from one core: plain function registration for scripts and serverless, and NestJS decorators for the framework case. The decorator marks a handler, the SDK exposes the route, handles ack/async/callback and idempotency, and completes when the handler resolves — so a customer writes an ordinary async method and gets durable orchestration. + +Ship with a working example that survives an engine restart mid-call. That example is the README. + +### B-07 · SDK: PHP / Laravel +**M · depends on B-02, B-05** + +Same contract, framework-idiomatic: service provider, route macro or attribute, queue integration for the async path since Laravel shops already run queues. This is the SDK that proves the protocol is not TypeScript-shaped, and Laravel is a large, underserved population for durable orchestration. + +### B-08 · Worker registration lifecycle +**S–M · depends on B-01** + +Register, health, deregister, drain. What happens when a worker is unreachable — how long a step waits, what the error taxonomy says, whether the run fails or parks. Also multiple workers behind one capability, and versioned capability rollout without breaking in-flight runs (interacts with A-05). + +### B-09 · Guard sub-workflow recursion +**S** + +Children spawn through the same `WorkflowSupervisorName` route as any run — no depth counter, no ancestry check, no cycle detection. A self-referencing schema is an unbounded fork bomb. Once agents compose workflows from data, this becomes triggerable **by model output**, not by a badly written schema. Depth counter in `SubWorkflowRef`, configurable max, ancestry cycle detection at spawn. + +### B-10 · Pass input into sub-workflows +**S · depends on A-01** + +`handleSubWorkflowAction` accepts and journals `input` and never passes it to the child. Wire it. Also add an output selector: the completion message carries the child's **entire** unfiltered per-node output map (`AggregatedOutputSnapshot()`), which balloons the parent's aggregated output and compounds A-09. + +Both are prerequisites for an agent composing and running a sub-workflow with parameters. + +--- + +## Tier C — Agents in Core + +### C-01 · Agent node ↔ registry tool binding +**M · depends on B-03** + +`ai/agent` and `ai/chat` exist as async functions. Bind them to registry capabilities so an agent's tool set is declared, versioned, and traced identically to a node call. An agent invoking a capability must be indistinguishable, in the journal and in the trace, from a graph node invoking it. + +### C-02 · Memory as a first-class concept +**M · overlaps A-11(a) — do them together** + +Working (run-scoped), episodic (past runs and conversations), semantic (durable facts). `ai/agent`'s `maxContextTokens`/`contextStrategy` trims inside one node execution and there is no graph-level equivalent, so conversation state grows unbounded in `aggregatedOutput` and in every journal payload. + +Semantic memory in markdown: human-readable, diffable, reviewable, auditable. An enterprise buyer can read what the system believes; a vector blob cannot be audited. Writes are proposals by default, resolved by policy or by a human gate. + +### C-03 · Model gateway +**M** + +Provider-agnostic routing: OpenRouter as the default breadth play, Azure OpenAI / Bedrock / vLLM for self-hosted enterprise. Policy per node — `{primary, fallback[], maxCost, maxLatency}` — with budget enforcement and redaction at the boundary before anything reaches a telemetry sink. + +Self-hosted models are a launch requirement for the regulated customers in the pipeline, not a roadmap item. + +### C-04 · GenAI telemetry and cost +**M · depends on A-08, C-03** + +Spans exist (`workflow.execute`, `node.execute`) with `workflow.*`/`node.*` attributes. Missing: agent id, model, tokens in/out, cost, gate decision, tenant. LLM usage is Prometheus-only (`ai/usage.go`), disconnected from the trace. + +Follow OTel GenAI conventions where they exist; namespace the rest. Meter cost at the gateway, tagged run and step, so "what did this customer's Tuesday cost" is a query. + +### C-05 · Human gate payload schema +**M · the Core half of the console's dynamic UI** + +A gate is not a string. Define in Core the typed payload a gate carries — options (single/multi), confirm with summary, form with schema — and the correction semantics where rejecting one item of a multi-item confirmation re-opens **only that item** and preserves the rest without restarting the run. + +Core owns the schema and the state transitions; Enterprise owns rendering (D-02). This split is what lets a customer build their own frontend against the same contract. + +Depends on A-11 being resolved, since correction requires addressable per-turn state. + +### C-06 · Durable timers and expiry sweeper +**M** + +Sleeps, awakeable timeouts and workflow timeouts are per-actor `SendAfter` — lost on restart, with no sweeper (the only periodic actors are `workflow_claim_actor.go` and `cron_scheduler.go`). A 72-hour gate that outlives a deploy has no deadline. The `Awakeable` row already carries `DeadlineAt`. Periodic sweep for expired pending awakeables and sleeps, re-arm live timers on boot, claim-aware so two nodes do not fire the same timeout. + +### C-07 · Expose awakeable IDs +**S** + +Minted in-engine, written only into a journal entry's `Data` map. `GET /v1/workflows/{id}` returns `{workflowId, status}`; `SnapshotTimelineEvent` drops `Data`; snapshot and trace persist only at terminal state. `AwakeableRepository.FindPending(workflowID)` exists with no route in `mux_worker.go`. A caller cannot learn the ID it must call back with — a documented feature no external caller can use. + +### C-08 · ForEach state survives restart, and stops silently lying +**M** + +Iteration bookkeeping lives in `forEachStates`/`iterThreadToForEach` — plain Go maps, never reconstructed — so an in-flight ForEach does not survive a restart, even though the journal already writes four foreach entry types and reads none. Same class of bug as A-02. + +Worse: `spawnForEachBatch` dispatches straight to the function pool, **bypassing system-function interception**, so `system/subworkflow`, `system/sleep` and `system/wait` inside a ForEach body hit the no-op placeholder, return success instantly, and silently do nothing. Silent wrong answers are worse than crashes. If the real fix is large, ship schema validation that **rejects** those functions inside a ForEach body in the meantime. + +### C-09 · Mid-flight run read model +**M** + +`GET /v1/workflows/{id}` returns `{workflowId, status}`. Snapshot and trace persist only in `sendWorkflowCompleted`. No SSE, no WebSocket; `memory_bus.go` is in-process and unreachable externally. + +With everything inside the engine, this is no longer optional: the console has nothing to render without it. Minimum: `GET /v1/workflows/{id}/state` built from the journal — thread positions, completed steps, pending steps, open gates. Streaming is a separate later task; do not conflate them. + +### C-10 · Per-node idempotency convention +**M · depends on B-01** + +Today `idempotency/store.go` is trigger-level only. Nothing derives a key from `(runID, execID, attempt)`, while the resume path re-executes pending steps — at-least-once with no dedup beneath it. `ExecutionInfo` already carries `WorkflowID` and `ExecID` into every function, so the raw material is there. Derive a stable key, put it in the protocol envelope, and let the SDKs dedupe on it transparently. + +### C-11 · Claim failure must fail closed +**S** + +`claimForThisNode` returns `true` on a claim-store error and runs the workflow anyway. Under Postgres pressure that is a split-brain path with side effects. Fail closed, log, retry next sweep. Also revisit `ClaimWorkflows(nodeID, 10)` every 5s — recovering 50 runs after a node loss takes ~25 seconds. + +### C-12 · Multi-thread resume correctness +**M · premise is inference · depends on F-02** + +`buildResumeAction` walks `lastCompletedThreadIDs` and returns the first non-noop action, so a multi-thread run that crashed between thread completions may lose its other branches. No test covers it either way. Verify with the F-02 multi-thread scenario first; if it reproduces, resume must fan out to every pending thread. + +--- + +## Tier D — Enterprise layer + +Separate repo, separate release cadence, consumes Core's public API only. Starts once Tier A is green — building a console on an engine whose durability is unverified means debugging both at once. + +- **D-01 · Tenancy, RBAC, SSO.** Row-level isolation enforced at the data layer. Approval rights distinct from execution rights. +- **D-02 · Console.** Schema editor, run inspector, gate approval rendering the C-05 payloads, trace rendered *as the workflow graph* so building and debugging share one mental model. This replaces the portal already started. +- **D-03 · Collaboration.** Multiple humans and agents on one run, server-authoritative sync, gate claiming so two people cannot approve the same thing twice, actor attribution on every action. +- **D-04 · Cost and usage dashboards.** Consumes C-04. +- **D-05 · Audit export.** Immutable log of runs, decisions, approvals, memory writes. +- **D-06 · Packaging.** Compose for pilot, Helm for production. Core's floor is already right — `CLUSTER_ENABLED=false`, `HA_ENABLED=false`, Postgres plus a writable path, embedded migrations, etcd and Redis optional. Do not regress it. + +--- + +## Found during verification — unfiled + +Raised by F-01 verification (2026-08-11), per "do not fix bugs found along the way". Not yet assigned +ids or tiers — that is the maintainer's call. Each says plainly whether it was **executed** or **read**. + +### V-a · Agent tool calls are invisible to OTel *and* Prometheus +**Read, from control flow · relevant to C-04 and to B-03's parity requirement** + +`internal/packages/agent_tools.go:86` (`InvokeTool`) calls `LoadedPackage.ExecuteFunctionSync` +(`internal/packages/loaded_package.go:50`), which goes straight to the transport and never enters +`internal/actors/workflow_func.go`. That file is the **only** creator of the `node.execute` span +(`:90`) and the only caller of `recordNodeDuration` (`:181`). So every tool an agent calls produces no +span and no `fuse_node_exec_duration_seconds` sample. + +This is direct evidence on B-03's hard design constraint — "an agent calling a capability and a node +calling the same capability must produce the same journal entries, the same span shape and the same +idempotency behaviour". Today they demonstrably do not. Distinct from the known gap that intercepted +`system/*` functions have no node span. + +### V-b · Resume appends a second `subworkflow:started` for one exec id, and nothing catches it +**Executed (`internal/actors/f01_2_verification_test.go`) · falls out of F-01.2, fix belongs with A-02** + +After a resume the journal holds two `subworkflow:started` entries for a single exec id with *different* +`childWorkflowId` — `internal/actors/workflow_handler.go:917-926` appends unconditionally with no read +of an existing entry. Any future replay handler for `subworkflow:started` must therefore tolerate or +repair pre-existing duplicates rather than assume one entry per exec. + +Postgres would not catch it either: `sub_workflow_refs` is UNIQUE on `child_workflow_id` only — there is +no unique index on the parent exec. + +### V-c · `Workflow.Next` nil-derefs on a missing node, crashing the run +**Executed (`TestWorkflow_Next_AfterLiveSchemaEditDropsInFlightNode_MustNotPanic`)** + +`internal/workflow/workflow.go:312` discards the error from `graph.FindNode` (`currentNode, _ := ...`) +and `:314` immediately dereferences the result, panicking in `internal/workflow/node.go:68` with +`runtime error: invalid memory address or nil pointer dereference`. + +Any missing-node condition — a live schema edit, replay against a changed schema, a bad audit entry — +takes down the workflow actor instead of failing the run. A-05 (version pinning) and A-06 (deep copy) +each remove *a* way to reach it; neither makes `Next` safe. Cheap to fix independently. + +--- + +## Dispatch order + +``` +F-01 ─┬─ F-02 ──┬── A-02, A-03, C-12 (need the harness to define "correct") + └─ A-06 │ +F-03 ────────────┤ + +A-07 ── independent, do first (compounds every restart) +A-01 ── independent → B-10 +A-04 ────────────────────────→ B-02 +A-05, A-08, A-09, A-10 ── independent +A-11 ── ADR before code, overlaps C-02 + +B-01 ──┬── B-02 ──┬── B-06, B-07 + ├── B-03 ──┴── B-04, C-01 + ├── B-05 (before the first SDK) + ├── B-08 + └── C-10 + +D-* ── after Tier A is green +``` + +**Sprint 1 — prove the engine.** F-01, F-02, F-03, A-07, A-06. Nothing user-visible ships; everything after becomes measurable. + +**Sprint 2 — write the spec.** B-01, plus A-01, A-04, A-05 in parallel. The spec is a document and a decision, not code, and it gates the entire B tier — do not let it slip behind implementation. + +**Sprint 3 — the pillar.** B-02, B-05, then B-06. One SDK, one conformance suite, one example that survives a restart mid-call. + +**Gate — do not choose an agent execution model before F-02 is green.** The report says cycles are genuinely supported, sub-workflow spawn works, and awakeables persist. Two of those three already carry known restart defects. Choosing on the strength of prose is how you end up with an elegant model resting on a primitive that has never been executed under SIGKILL. + +--- + +## Rules for the implementing agent + +- **Verify before fixing.** Every task cites a report, not the code. If the premise does not hold, close it and report it. +- **One task, one PR.** They are deliberately separable. +- **A test reproducing the bug lands before the fix**, in the same PR, failing in the first commit. +- **Do not fix bugs found along the way.** Add them here with evidence and keep going. +- **The protocol spec is a contract.** Once B-01 is published and an SDK exists, changing it breaks strangers. Treat it with more care than the engine internals. +- **Core stays usable without Enterprise.** If a Core feature only makes sense with the commercial layer, it belongs in the commercial layer. +- **Public API changes need a changelog entry and a migration path.** Apache-2.0, with users who are not you. diff --git a/docs/adr/0034-backlog-delivery-pipeline-of-subagents.md b/docs/adr/0034-backlog-delivery-pipeline-of-subagents.md new file mode 100644 index 0000000..b8400ef --- /dev/null +++ b/docs/adr/0034-backlog-delivery-pipeline-of-subagents.md @@ -0,0 +1,95 @@ +# 0034. Deliver backlog work through a pipeline of specialised subagents + +- Status: Accepted +- Date: 2026-08-11 +- Deciders: FUSE maintainers + +## Context and Problem Statement + +The work queue (`BACKLOG_V2.md`) has a property that ordinary backlogs do not: **its tasks were +derived from a capability report about the code, not from the code.** The document says so in its +own header — "The report describes the code; it is not the code" — and task F-01 exists purely to +verify three findings before anything is built. At least one premise is already suspect: A-03 +describes a restart re-invoking a worker "with a new execID", while `replayPendingThread` +(`internal/workflow/workflow.go`) appears to reuse the pending step's existing execID. + +An agent that treats such a backlog as ground truth will implement around bugs that do not exist, +and will do it confidently. The failure is silent and expensive. + +Two further properties raise the cost of getting this wrong. Tier B makes a **published protocol +spec** the product pillar — once an SDK exists, a wire change breaks strangers, so spec work must +precede and outrank implementation. And F-02 (a real-process crash-resume harness) **blocks all of +Tier A** while gating the agent-execution-model decision, so "correct" is undefined until it exists. + +How should engine work be executed so that verification, planning, implementation and review are +distinct, auditable steps rather than one model's single pass? + +## Decision Drivers + +- A premise that does not reproduce must **close** its task — that outcome needs to be as easy to + produce as a fix, and as respected. +- Separation of powers: the actor that reproduces a bug should not be the actor that fixes it, and + neither should be the actor that judges the fix. +- Engine knowledge (18 journal types, 5 replayed; driver parity; pool size 3) is expensive to + rediscover per session and must be written down once, citation-grounded. +- Protocol changes need a different craft — and a different gate — from Go changes. +- Portability: whatever we add must live under `.agents/` per [ADR-0009](0009-portable-ai-agent-guidance.md). + +## Considered Options + +- **One general agent plus the existing rules.** Zero new surface; relies on `.agents/rules/` + already being loaded. But nothing forces verification before implementation, and nothing stops a + wire-surface change from landing without a spec. +- **A skill per backlog tier.** Knowledge packs only, no role separation — the same context that + wrote the code also reviews it, which is the review failure mode we are trying to avoid. +- **A pipeline of specialised subagents plus knowledge packs** (chosen). + +## Decision Outcome + +Chosen option: **seven subagents under `.agents/agents/`, orchestrated by a `backlog-task` skill, +backed by seven engine knowledge packs under `.agents/skills/`.** + +The pipeline is: locate & scope → **verify premise** → research → **plan** → implement → **review** +→ gates & land. Each stage is a separate agent with its own tool allowlist and explicit boundaries: + +| Agent | Owns | Must not | +| --- | --- | --- | +| `fuse-premise-verifier` | Reproduces a claim with an executed failing test | Fix anything | +| `fuse-engine-researcher` | Read-only maps, every claim CONFIRMED vs INFERRED | Write any file | +| `fuse-implementation-planner` | Files, 3-level test plan, parity, migration, ADR call | Plan an unverified premise | +| `fuse-protocol-spec-author` | B-01 spec + ADR (Markdown only) | Write engine Go | +| `fuse-go-implementer` | Test-first implementation, runs the gates | Bundle tasks; drive-by fixes | +| `fuse-e2e-harness-engineer` | F-02 real-process SIGKILL harness | Fix engine bugs it finds | +| `fuse-code-reviewer` | Adversarial pre-merge review | Edit the code | + +Two boundaries are drawn explicitly because they are the ones dispatch gets wrong: a test that +kills and restarts a real OS process belongs to `fuse-e2e-harness-engineer` and involves no +production Go; everything else, including all engine fixes, belongs to `fuse-go-implementer`. And +`fuse-protocol-spec-author` is the *first* call for B-01 or any wire-surface change — ahead of the +planner, which plans Go only against a spec clause that already exists. + +`.claude/agents` symlinks into `.agents/agents/`, matching the existing wiring for rules, skills and +commands. + +### Consequences + +- Good: "the premise did not reproduce" becomes a first-class, cheap outcome instead of an + awkward one, which is what the backlog's own rules demand. +- Good: engine knowledge is written down once, with file:symbol citations, and each pack names the + backlog task that owns each known defect — so no agent mistakes a tracked bug for the contract. +- Good: a wire-surface change cannot reach code without passing through a spec author. +- Bad: more surface to keep current. A pack that drifts is worse than no pack, because it is + trusted. Packs cite symbols as well as line numbers so a stale offset degrades to a re-grep + rather than a wrong answer. +- Neutral: the pipeline is advisory. A maintainer can invoke any agent directly. + +## More Information + +- Entry point: `.agents/skills/backlog-task/SKILL.md` (`/backlog-task F-02`), which carries a + routing table for every backlog id with a premise-confidence column. +- Knowledge packs: `durable-execution-internals`, `crash-resume-testing`, + `persistence-and-migrations`, `observability-tracing`, `function-package-authoring`, + `remote-node-protocol`, `capability-registry-and-agents`. +- Layout and symlink rationale: [ADR-0009](0009-portable-ai-agent-guidance.md). +- The durability contract these packs describe: [ADR-0010](0010-durable-execution-journal-and-replay.md). +- Tier D (Enterprise) is out of scope here: it is a separate repo consuming Core's public API. diff --git a/docs/adr/README.md b/docs/adr/README.md index 0a29574..945f123 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -60,6 +60,7 @@ copying [`template.md`](template.md). | 0031 | [Settings, secrets & environments: a SecretStore seam](0031-settings-secrets-and-environments.md) | Accepted | 2026-06-02 | | 0032 | [Sub-workflow composition: child workflows as first-class instances](0032-sub-workflow-composition.md) | Accepted | 2026-06-03 | | 0033 | [Dependency injection & app composition with uber-go/fx](0033-dependency-injection-and-app-composition.md) | Accepted | 2026-06-03 | +| 0034 | [Deliver backlog work through a pipeline of specialised subagents](0034-backlog-delivery-pipeline-of-subagents.md) | Accepted | 2026-08-11 | ### Proposed backlog (not yet implemented) diff --git a/internal/actors/f01_1_verification_test.go b/internal/actors/f01_1_verification_test.go new file mode 100644 index 0000000..3cc5a4c --- /dev/null +++ b/internal/actors/f01_1_verification_test.go @@ -0,0 +1,565 @@ +package actors + +// f01_1_verification_test.go — verification of BACKLOG_V2.md F-01, finding 1 ("f01-1-span"): +// +// "internal/actors/workflow_func.go — node span context discarded (`_ = nodeCtx`), so function +// code cannot create child spans or inject `traceparent` outbound. Now critical, because it is +// the seam every SDK crosses." +// +// WHAT THESE TESTS PROVE +// - The defect test and the actor control drive the REAL production entry point, +// WorkflowFunc.HandleMessage, with a stub +// gen.Process (Send + Log only) — the same technique the sibling verifier uses +// (f01_2_verification_test.go). Nothing is mirrored or re-implemented: the ExecutionInfo the +// node function receives is the one workflow_func.go builds at :139, and the node span is the +// one it starts at :90. +// - The defect test proves the observable consequence: a node function handed a live +// *workflow.ExecutionInfo, during an execution whose inbound message carries a sampled +// traceparent, has NO way to recover the trace context of its own node span, and therefore +// cannot continue the run's trace in an outbound call. The probe is deliberately shape-blind +// (see f01ReachTraceFrom): it accepts a context.Context, a trace.Span, a trace.SpanContext, a +// map[string]string carrier, a traceparent string, or a "traceparent" key in the function +// input, reached via any exported field or any zero-argument accessor. Any fix that puts the +// node span's context within reach of function code flips it green; no particular field name, +// type or fix shape is assumed. +// - The two control tests PASS today, and pin down that the red is not an artifact. The probe +// control feeds the probe a live node span context through five different post-fix shapes +// (context field, context accessor, carrier field, span field, traceparent string) and asserts +// it recovers the node span every time, plus a negative case that yields nothing — so "goes +// green under any legitimate fix" is executed, not merely argued. The actor control proves the +// harness really executes the registered function through the package/transport path and really +// replies to the workflow handler, and this same provider + this same inbound carrier DO +// produce an injectable node-span context when the extract/StartSpan pair of +// workflow_func.go:89-95 is performed. (With OTEL_ENABLED=false, tracing.noopProvider() has an +// empty composite propagator and InjectCarrier always returns an empty map — provider.go:82-87 +// — so the enabled path is required for this to mean anything.) +// +// WHAT THEY DO NOT PROVE +// - Nothing about the HTTP boundary (inbound extraction, A-04 break 1), about async completion +// spans, or about the persisted ExecutionTrace (a different notion of "trace", ADR-0020). +// - Nothing about ergo mailbox ordering, supervision or Init: HandleMessage is called directly. +// - No repository is involved, so there is no memory/Postgres driver dimension here. +// - They do not assert on source text. Sub-claim "(a) the literal `_ = nodeCtx` is present" is +// deliberately NOT asserted: any no-op consumption of nodeCtx would satisfy a source-text +// check while leaving the defect in place, so the behavioural consequence is asserted instead. +// - One fix shape cannot be absorbed by any test: changing the workflow.Function signature +// itself (e.g. func(context.Context, *ExecutionInfo)) breaks compilation of every function +// literal in the repo, this file included. That is a mechanical update, not a weakening. +// +// The defect test asserts the DESIRED behaviour, so it is expected to be RED on main. + +import ( + "context" + "reflect" + "strings" + "testing" + "time" + + "ergo.services/ergo/gen" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/trace" + + "github.com/open-source-cloud/fuse/internal/actors/actornames" + "github.com/open-source-cloud/fuse/internal/app/config" + "github.com/open-source-cloud/fuse/internal/messaging" + "github.com/open-source-cloud/fuse/internal/metrics" + "github.com/open-source-cloud/fuse/internal/packages" + "github.com/open-source-cloud/fuse/internal/packages/transport" + "github.com/open-source-cloud/fuse/internal/tracing" + "github.com/open-source-cloud/fuse/internal/workflow/workflowactions" + "github.com/open-source-cloud/fuse/pkg/workflow" +) + +const ( + // A fixed, SAMPLED W3C traceparent, as an SDK caller would send it. Sampled matters: with an + // unsampled parent the SDK returns a non-recording span that REUSES the parent span context, + // and the "new engine span id" assertion below would be meaningless. + f01InboundTraceparent = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" + f01InboundTraceID = "4bf92f3577b34da6a3ce929d0e0e4736" + f01InboundSpanID = "00f067aa0ba902b7" + + f01TraceparentKey = "traceparent" + f01HexDigits = "0123456789abcdef" + + f01PackageID = "fuse/pkg/f01" + f01FunctionName = "outbound" + f01FullFunctionID = f01PackageID + "/" + f01FunctionName +) + +// --- harness ----------------------------------------------------------------------------------- + +// f011Log is a no-op gen.Log. Only the levels WorkflowFunc.HandleMessage uses are implemented; any +// other method would panic on the nil embedded interface, which is the intended loud failure. +type f011Log struct { + gen.Log +} + +func (l *f011Log) Trace(_ string, _ ...any) {} +func (l *f011Log) Debug(_ string, _ ...any) {} +func (l *f011Log) Info(_ string, _ ...any) {} +func (l *f011Log) Warning(_ string, _ ...any) {} +func (l *f011Log) Error(_ string, _ ...any) {} +func (l *f011Log) Panic(_ string, _ ...any) {} + +// f011Sent records one outbound send made by the actor under test. +type f011Sent struct { + to any + message any +} + +// f011Process is a stub gen.Process that records outbound sends instead of delivering them. +type f011Process struct { + gen.Process + log *f011Log + sent []f011Sent +} + +func (p *f011Process) Send(to any, message any) error { + p.sent = append(p.sent, f011Sent{to: to, message: message}) + return nil +} + +func (p *f011Process) Log() gen.Log { return p.log } + +// f011Registry is an isolated packages.Registry holding exactly one package, so the test never +// touches the process-wide singleton returned by packages.NewPackageRegistry(). +type f011Registry struct { + pkg *packages.LoadedPackage +} + +func (r *f011Registry) Register(_ *workflow.Package) {} + +func (r *f011Registry) Get(pkgID string) (*packages.LoadedPackage, error) { + if r.pkg != nil && r.pkg.ID == pkgID { + return r.pkg, nil + } + return nil, packages.ErrLoadedPackageNotFound +} + +func (r *f011Registry) Has(pkgID string) bool { + return r.pkg != nil && r.pkg.ID == pkgID +} + +func (r *f011Registry) List() ([]*packages.LoadedPackage, error) { + if r.pkg == nil { + return []*packages.LoadedPackage{}, nil + } + return []*packages.LoadedPackage{r.pkg}, nil +} + +// f01Provider builds a REAL, OTel-enabled tracing.Provider. tracing.Provider is a concrete struct +// with unexported fields and no seam, so an enabled provider can only be obtained from +// tracing.NewProvider — which also installs itself into the OTel globals (provider.go:72-73). +// Those globals are captured and restored so this file (which sorts first in the package) does not +// hand every later test a shut-down provider. +func f01Provider(t *testing.T) *tracing.Provider { + t.Helper() + + prevTracerProvider := otel.GetTracerProvider() + prevPropagator := otel.GetTextMapPropagator() + + cfg := &config.Config{} + cfg.Otel.Enabled = true + cfg.Otel.Endpoint = "localhost:4317" + cfg.Otel.Insecure = true + cfg.Otel.ServiceName = "fuse-f01-verification" + cfg.Otel.ServiceVersion = "test" + + provider, err := tracing.NewProvider(cfg) + require.NoError(t, err) + require.NotNil(t, provider) + + t.Cleanup(func() { + otel.SetTracerProvider(prevTracerProvider) + otel.SetTextMapPropagator(prevPropagator) + // No collector listens on the endpoint; keep the flush short so the suite does not pay the + // full export timeout. Spans are never read back — every assertion here works off the + // propagator, not off exported spans. + ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) + defer cancel() + _ = provider.Shutdown(ctx) + }) + + return provider +} + +// f01Metadata is the minimum metadata for an internal, code-backed function. Concurrency and +// RateLimit are intentionally absent: MapToRegistryPackage never sets them, which is why +// workflow_func.go:102/:108 never dereference the (nil) concurrency manager and rate limiter. +func f01Metadata() workflow.FunctionMetadata { + return workflow.FunctionMetadata{ + Transport: transport.Internal, + Input: workflow.InputMetadata{ + Parameters: make([]workflow.ParameterSchema, 0), + Edges: workflow.InputEdgeMetadata{Parameters: make([]workflow.ParameterSchema, 0)}, + }, + Output: workflow.OutputMetadata{ + Parameters: make([]workflow.ParameterSchema, 0), + Edges: make([]workflow.OutputEdgeMetadata, 0), + }, + } +} + +// f01NewWorkflowFunc wires a real WorkflowFunc worker around fn, with a real OTel-enabled provider +// and a real metrics registry, backed by a stub gen.Process. +func f01NewWorkflowFunc(t *testing.T, provider *tracing.Provider, fn workflow.Function) (*WorkflowFunc, *f011Process) { + t.Helper() + + pkg := packages.MapToRegistryPackage( + workflow.NewPackage(f01PackageID, workflow.NewFunction(f01FunctionName, f01Metadata(), fn)), + ) + loadedFn, ok := pkg.Functions[f01FullFunctionID] + require.Truef(t, ok, "package %s must expose %s", f01PackageID, f01FullFunctionID) + require.NotNil(t, loadedFn.Transport, "the function must be registered as code-backed (executable)") + + worker := &WorkflowFunc{ + packageRegistry: &f011Registry{pkg: pkg}, + fuseMetrics: metrics.NewFuseMetrics(), + tracingProvider: provider, + } + process := &f011Process{log: &f011Log{}} + // Promoted from the embedded act.Actor: this is the gen.Process the worker sends and logs through. + worker.Process = process + + return worker, process +} + +// f01ExecuteMessage builds the message WorkflowHandler sends to the func pool +// (messaging.NewExecuteFunctionMessage, as called from workflow_handler.go), carrying an inbound +// trace carrier. +func f01ExecuteMessage(wfID workflow.ID, execID workflow.ExecID) messaging.Message { + return messaging.NewExecuteFunctionMessage( + wfID, + &workflowactions.RunFunctionAction{ + ThreadID: 1, + FunctionID: f01FullFunctionID, + FunctionExecID: execID, + Args: map[string]any{}, + }, + "", + map[string]string{f01TraceparentKey: f01InboundTraceparent}, + ) +} + +// --- shape-blind trace probe ------------------------------------------------------------------- + +// f01Reach is what a node function could recover about its own trace context from the only value +// it is handed. +type f01Reach struct { + Source string // e.g. `field Ctx`, `method Context()`, `input["traceparent"]`; empty = nothing + TraceID string + SpanID string + Scanned []string +} + +type f01Candidate struct { + name string + value reflect.Value +} + +// f01ReachTraceFrom asks the question the backlog finding is about — "can function code continue +// the run's trace?" — without assuming how a fix would answer it. It walks every exported field of +// the struct behind the pointer, every zero-argument accessor that returns a plausible trace +// carrier, and (for a real ExecutionInfo) the function input, and returns the first one that yields +// a usable W3C trace context. +// +// It takes any so TestF01TraceProbe_FindsEveryFixShape can prove the probe is not itself the reason +// the defect test is red. +func f01ReachTraceFrom(provider *tracing.Provider, info any) f01Reach { + reach := f01Reach{Scanned: make([]string, 0, 8)} + if info == nil { + return reach + } + if v := reflect.ValueOf(info); v.Kind() != reflect.Ptr || v.IsNil() || v.Elem().Kind() != reflect.Struct { + return reach + } + + for _, candidate := range f01TraceCandidates(info) { + reach.Scanned = append(reach.Scanned, candidate.name) + traceID, spanID, ok := f01TraceIDsOf(provider, candidate.value) + if !ok { + continue + } + reach.Source, reach.TraceID, reach.SpanID = candidate.name, traceID, spanID + return reach + } + + return reach +} + +func f01TraceCandidates(info any) []f01Candidate { + value := reflect.ValueOf(info).Elem() + structType := value.Type() + ptrType := reflect.TypeOf(info) + ptrValue := reflect.ValueOf(info) + + candidates := make([]f01Candidate, 0, structType.NumField()+ptrType.NumMethod()+1) + for i := range structType.NumField() { + field := structType.Field(i) + if !field.IsExported() { + continue + } + candidates = append(candidates, f01Candidate{name: "field " + field.Name, value: value.Field(i)}) + } + for i := range ptrType.NumMethod() { + method := ptrType.Method(i) + if method.Type.NumIn() != 1 || method.Type.NumOut() != 1 || !f01IsTraceCarrierType(method.Type.Out(0)) { + continue + } + candidates = append(candidates, f01Candidate{ + name: "method " + method.Name + "()", + value: ptrValue.Method(i).Call(nil)[0], + }) + } + // A carrier smuggled through the function input is a legitimate fix shape too. + if exec, isExecInfo := info.(*workflow.ExecutionInfo); isExecInfo && exec.Input != nil { + candidates = append(candidates, f01Candidate{ + name: `input["` + f01TraceparentKey + `"]`, + value: reflect.ValueOf(exec.Input.GetStr(f01TraceparentKey)), + }) + } + + return candidates +} + +func f01IsTraceCarrierType(t reflect.Type) bool { + ctxType := reflect.TypeOf((*context.Context)(nil)).Elem() + spanType := reflect.TypeOf((*trace.Span)(nil)).Elem() + + switch { + case t == ctxType || t.Implements(ctxType): + return true + case t == spanType || t.Implements(spanType): + return true + case t == reflect.TypeOf(trace.SpanContext{}): + return true + case t == reflect.TypeOf(map[string]string{}): + return true + case t.Kind() == reflect.String: + return true + default: + return false + } +} + +// f01TraceIDsOf turns one candidate into (traceID, spanID) if it carries a usable trace context. +func f01TraceIDsOf(provider *tracing.Provider, value reflect.Value) (string, string, bool) { + if !value.IsValid() { + return "", "", false + } + switch value.Kind() { + case reflect.Interface, reflect.Map, reflect.Ptr, reflect.Func, reflect.Slice: + if value.IsNil() { + return "", "", false + } + case reflect.String: + return f01ParseTraceparent(value.String()) + default: + } + + switch carrier := value.Interface().(type) { + case context.Context: + return f01ParseTraceparent(provider.InjectCarrier(carrier)[f01TraceparentKey]) + case trace.Span: + ctx := trace.ContextWithSpan(context.Background(), carrier) + return f01ParseTraceparent(provider.InjectCarrier(ctx)[f01TraceparentKey]) + case trace.SpanContext: + ctx := trace.ContextWithSpanContext(context.Background(), carrier) + return f01ParseTraceparent(provider.InjectCarrier(ctx)[f01TraceparentKey]) + case map[string]string: + return f01ParseTraceparent(carrier[f01TraceparentKey]) + default: + return "", "", false + } +} + +// f01ParseTraceparent extracts (traceID, spanID) from a W3C traceparent header value. +func f01ParseTraceparent(traceparent string) (string, string, bool) { + parts := strings.Split(traceparent, "-") + if len(parts) != 4 { + return "", "", false + } + traceID, spanID := parts[1], parts[2] + if len(traceID) != 32 || len(spanID) != 16 { + return "", "", false + } + if strings.Trim(traceID, f01HexDigits) != "" || strings.Trim(spanID, f01HexDigits) != "" { + return "", "", false + } + if strings.Trim(traceID, "0") == "" || strings.Trim(spanID, "0") == "" { + return "", "", false + } + return traceID, spanID, true +} + +// --- post-fix ExecutionInfo shapes, used only by the probe control ------------------------------ + +type f01CtxFieldShape struct { + WorkflowID workflow.ID + Ctx context.Context //nolint:containedctx // models a post-fix ExecutionInfo field +} + +type f01AccessorShape struct { + WorkflowID workflow.ID + ctx context.Context //nolint:containedctx // models a post-fix ExecutionInfo field +} + +func (i *f01AccessorShape) Context() context.Context { return i.ctx } + +type f01CarrierFieldShape struct { + WorkflowID workflow.ID + TraceCarrier map[string]string +} + +type f01SpanFieldShape struct { + WorkflowID workflow.ID + Span trace.Span +} + +type f01TraceparentFieldShape struct { + WorkflowID workflow.ID + Traceparent string +} + +type f01NoTraceShape struct { + WorkflowID workflow.ID + Environment string + Ctx context.Context //nolint:containedctx // deliberately nil: the pre-fix situation + Tags map[string]string +} + +// --- tests ------------------------------------------------------------------------------------- + +// Control. Expected GREEN: it proves the harness drives the production path end to end AND that +// this provider can continue the inbound trace, so the red below is the missing seam and not a +// dead propagator or a function that never ran. +func TestWorkflowFunc_HandleMessage_RunsNodeFunctionAndProviderContinuesInboundTrace(t *testing.T) { + // Arrange + provider := f01Provider(t) + wfID := workflow.NewID() + execID := workflow.NewExecID(1) + + seen := make([]*workflow.ExecutionInfo, 0, 1) + worker, process := f01NewWorkflowFunc(t, provider, func(info *workflow.ExecutionInfo) (workflow.FunctionResult, error) { + seen = append(seen, info) + return workflow.NewFunctionResultSuccess(), nil + }) + + // Act + err := worker.HandleMessage(gen.PID{}, f01ExecuteMessage(wfID, execID)) + + // Assert — the real production path ran the registered function and answered the handler. + require.NoError(t, err) + require.Len(t, seen, 1, "control: HandleMessage must have executed the registered node function exactly once") + assert.Equal(t, wfID, seen[0].WorkflowID, "control: the function receives the message's workflow id") + assert.Equal(t, execID, seen[0].ExecID, "control: the function receives the message's exec id") + + require.Len(t, process.sent, 1, "control: exactly one message must go back to the workflow handler") + assert.Equal(t, actornames.WorkflowHandlerName(wfID), process.sent[0].to) + resultMsg, isMessage := process.sent[0].message.(messaging.Message) + require.True(t, isMessage, "control: the handler reply must be a messaging.Message") + assert.Equal(t, messaging.FunctionResult, resultMsg.Type) + + // Assert — the provider is capable: the extract/StartSpan pair of workflow_func.go:89-95, run + // with this provider and this carrier, yields an injectable node-span context. + parentCtx := provider.ExtractCarrier(context.Background(), map[string]string{f01TraceparentKey: f01InboundTraceparent}) + nodeCtx, nodeSpan := provider.StartSpan(parentCtx, "node.execute") + defer nodeSpan.End() + + traceID, spanID, ok := f01ParseTraceparent(provider.InjectCarrier(nodeCtx)[f01TraceparentKey]) + require.True(t, ok, "control: an enabled provider must inject a traceparent from a node span context") + assert.Equal(t, f01InboundTraceID, traceID, "control: the node span must continue the inbound trace") + assert.NotEqual(t, f01InboundSpanID, spanID, "control: the node span must have its own span id") +} + +// Control. Expected GREEN: the probe the defect test uses recognises every fix shape anyone might +// reasonably choose for threading the node span context to function code, and produces no false +// positive from a struct that carries none. Without this, a red below would be ambiguous between +// "the seam is missing" and "the probe only knows one fix shape". +func TestF01TraceProbe_FindsEveryFixShape(t *testing.T) { + // Arrange — a live node span context, exactly as workflow_func.go:90 produces one. + provider := f01Provider(t) + parentCtx := provider.ExtractCarrier(context.Background(), map[string]string{f01TraceparentKey: f01InboundTraceparent}) + nodeCtx, nodeSpan := provider.StartSpan(parentCtx, "node.execute") + defer nodeSpan.End() + nodeSpanID := nodeSpan.SpanContext().SpanID().String() + nodeCarrier := provider.InjectCarrier(nodeCtx) + + testCases := []struct { + name string + info any + wantSource string + }{ + {"context field", &f01CtxFieldShape{WorkflowID: workflow.NewID(), Ctx: nodeCtx}, "field Ctx"}, + {"context accessor", &f01AccessorShape{WorkflowID: workflow.NewID(), ctx: nodeCtx}, "method Context()"}, + {"carrier field", &f01CarrierFieldShape{WorkflowID: workflow.NewID(), TraceCarrier: nodeCarrier}, "field TraceCarrier"}, + {"span field", &f01SpanFieldShape{WorkflowID: workflow.NewID(), Span: nodeSpan}, "field Span"}, + { + "traceparent string field", + &f01TraceparentFieldShape{WorkflowID: workflow.NewID(), Traceparent: nodeCarrier[f01TraceparentKey]}, + "field Traceparent", + }, + } + + for _, testCase := range testCases { + t.Run(testCase.name, func(t *testing.T) { + // Act + reach := f01ReachTraceFrom(provider, testCase.info) + + // Assert + assert.Equal(t, testCase.wantSource, reach.Source, "scanned: %v", reach.Scanned) + assert.Equal(t, f01InboundTraceID, reach.TraceID, "the fix shape must expose the run's trace id") + assert.Equal(t, nodeSpanID, reach.SpanID, "the fix shape must expose the node span, not the caller's remote span") + assert.NotEqual(t, f01InboundSpanID, reach.SpanID) + }) + } + + t.Run("no trace anywhere", func(t *testing.T) { + // Act — a nil context field, an unrelated map and a uuid-shaped string must not be mistaken + // for a trace context. + reach := f01ReachTraceFrom(provider, &f01NoTraceShape{ + WorkflowID: workflow.NewID(), + Environment: "default", + Tags: map[string]string{"team": "core"}, + }) + + // Assert + assert.Empty(t, reach.Source) + assert.Empty(t, reach.TraceID) + assert.NotEmpty(t, reach.Scanned, "the probe must actually have walked the struct") + }) +} + +// The defect. Expected RED on main: workflow_func.go throws the node span context away, so the +// function it then invokes cannot parent anything on the node span nor continue the run's trace. +func TestWorkflowFunc_HandleMessage_GivesNodeFunctionTheNodeSpanContext(t *testing.T) { + // Arrange + provider := f01Provider(t) + calls := 0 + var reach f01Reach + worker, _ := f01NewWorkflowFunc(t, provider, func(info *workflow.ExecutionInfo) (workflow.FunctionResult, error) { + // What every outbound-HTTP / SDK node must be able to do: continue the run's trace. + calls++ + reach = f01ReachTraceFrom(provider, info) + return workflow.NewFunctionResultSuccess(), nil + }) + + // Act + err := worker.HandleMessage(gen.PID{}, f01ExecuteMessage(workflow.NewID(), workflow.NewExecID(1))) + + // Assert + require.NoError(t, err) + require.Equal(t, 1, calls, "the node function must have been executed exactly once") + + assert.NotEmptyf(t, reach.Source, + "node function has no reachable trace context on *workflow.ExecutionInfo (scanned: %v): the node span context created at workflow_func.go:90 is discarded, so function code can neither create a child span nor inject traceparent outbound", + reach.Scanned) + assert.Equalf(t, f01InboundTraceID, reach.TraceID, + "an outbound call from the node function must continue the run's trace %s (reached via %q, scanned %v)", + f01InboundTraceID, reach.Source, reach.Scanned) + assert.NotEqualf(t, f01InboundSpanID, reach.SpanID, + "the node function's context must be parented on the engine's node span, not on the caller's remote span (reached via %q)", + reach.Source) +} diff --git a/internal/actors/f01_2_verification_test.go b/internal/actors/f01_2_verification_test.go new file mode 100644 index 0000000..a2cc92a --- /dev/null +++ b/internal/actors/f01_2_verification_test.go @@ -0,0 +1,310 @@ +package actors + +// f01_2_verification_test.go — verification of BACKLOG_V2.md F-01, finding 2 ("f01-2-subworkflow"): +// +// "Restart with a pending `system/subworkflow` spawns a second child — `findPendingThreads` +// matches `step:started` without completion, `replayPendingThread` re-issues, interception +// fires again." +// +// WHAT THIS TEST PROVES +// - The OBSERVABLE defect only: after a restart of a run parked on a SYNCHRONOUS +// system/subworkflow, MORE THAN ONE DISTINCT child workflow instance exists for the single +// parent exec id that spawned it. Distinctness is collected from every place a child instance +// becomes visible outside the handler: the SubWorkflowRef rows, the durable journal's +// `subworkflow:started` entries, and the `workflow:trigger` messages the handler emitted. +// - It deliberately asserts NOTHING about the route taken to get there: not `findPendingThreads`, +// not the action `Resume()` returns, not which messages were sent, not how many repository rows +// were written. Those are all things a legitimate fix may leave exactly as they are (see +// "WHY THIS SHAPE" below), and asserting them would make this test red after a correct fix. +// +// WHAT THIS TEST DOES NOT PROVE +// - Nothing here dies. No SIGKILL, no process restart, no ergo runtime, no supervision, no +// mailbox ordering, no actor Init. The "restart" is modelled by rebuilding the aggregate from +// the persisted journal and running a second WorkflowHandler value over the same repositories, +// which is the shape of workflow_handler.go:150-159. Establishing that a real process death +// and restart produces the same outcome is F-02's job ("kill with a pending +// `system/subworkflow` (exactly one child after restart)"); until F-02 exists, this is engine +// logic evidence, not crash-resume evidence. +// - The handler is driven through a stub gen.Process (only Send and Log implemented), so the +// child workflow is never actually started by WorkflowSupervisor. The test models that side +// effect by inserting the child workflow row itself (see the Arrange block), because the +// pre-crash world the backlog describes is one where the child IS already running. +// - Memory repositories only. Postgres rebuilds the aggregate through +// postgres/workflow.go:45 + loadGraph and stores refs in a table with its own constraints; +// this test constructs the fresh aggregate by hand to match that shape but does not execute +// the Postgres driver. Note MemoryWorkflowRepository.SaveSubWorkflowRef +// (internal/repositories/workflow_memory.go:74-82) appends unconditionally with no dedup, which +// is precisely why this test counts DISTINCT child ids and not rows. +// +// SCOPE — READ, NOT EXECUTED +// The duplicate is asserted for a SYNCHRONOUS sub-workflow (async=false). Reading +// handleSubWorkflowAction, the async=true branch calls SetResultFor at workflow_handler.go:936, +// and SetResultFor (internal/workflow/workflow.go:421-441) appends a `step:completed` for that +// exec id, so findPendingThreads (workflow.go:267) would never match it on replay. That path was +// READ, not executed, and this file makes no claim about it. +// +// WHY THIS SHAPE (fix-agnosticism) +// There are at least two legitimate fix sites, and the assertion below goes green under both: +// (i) the aggregate declines to re-issue — findPendingThreads/replayPendingThread +// (workflow.go:267-306) or replayJournalEntries (workflow.go:199) learn to read +// `subworkflow:started`, so Resume() no longer returns the sub-workflow step; +// (ii) the actor stays idempotent — handleSubWorkflowAction (workflow_handler.go:901-902) +// reuses the child id already journalled for that ParentExecID instead of minting a new +// one with workflow.NewID(), and may still re-save the ref and re-send the trigger. +// Under (ii) the aggregate is untouched, Resume() still returns system/subworkflow, a second ref +// row is still written and a second trigger message is still sent — only the child IDENTITY +// collapses to one. That is why the count is over distinct ids from all three sources. + +import ( + "fmt" + "sort" + "testing" + + "ergo.services/ergo/gen" + "github.com/open-source-cloud/fuse/internal/app/config" + "github.com/open-source-cloud/fuse/internal/messaging" + "github.com/open-source-cloud/fuse/internal/packages" + "github.com/open-source-cloud/fuse/internal/packages/functions/system" + "github.com/open-source-cloud/fuse/internal/repositories" + internalworkflow "github.com/open-source-cloud/fuse/internal/workflow" + "github.com/open-source-cloud/fuse/internal/workflow/workflowactions" + "github.com/open-source-cloud/fuse/pkg/workflow" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const ( + f012ParentSchemaID = "f01-2-parent" + f012ChildSchemaID = "f01-2-child-schema" +) + +// f012Log is a no-op gen.Log. Only the levels the exercised code path uses are implemented; any +// other method would panic on the nil embedded interface, which is the intended loud failure. +type f012Log struct { + gen.Log +} + +func (l *f012Log) Trace(_ string, _ ...any) {} +func (l *f012Log) Debug(_ string, _ ...any) {} +func (l *f012Log) Info(_ string, _ ...any) {} +func (l *f012Log) Warning(_ string, _ ...any) {} +func (l *f012Log) Error(_ string, _ ...any) {} +func (l *f012Log) Panic(_ string, _ ...any) {} + +// f012Process is a stub gen.Process that records outbound sends instead of delivering them. +type f012Process struct { + gen.Process + log *f012Log + sent []f012Sent +} + +type f012Sent struct { + to any + message any +} + +func (p *f012Process) Send(to any, message any) error { + p.sent = append(p.sent, f012Sent{to: to, message: message}) + return nil +} + +func (p *f012Process) Log() gen.Log { return p.log } + +// f012ParentGraph builds n1 (ordinary function) -> n2 (system/subworkflow, schemaId mapped). +func f012ParentGraph(t *testing.T) *internalworkflow.Graph { + t.Helper() + schema := &internalworkflow.GraphSchema{ + ID: f012ParentSchemaID, + Name: "f01-2 parent workflow", + Nodes: []*internalworkflow.NodeSchema{ + {ID: "n1", Function: "debug/nil"}, + {ID: "n2", Function: system.SubWorkflowFullFunctionID}, + }, + Edges: []*internalworkflow.EdgeSchema{ + { + ID: "e1", + From: "n1", + To: "n2", + Input: []internalworkflow.InputMapping{ + {Source: internalworkflow.SourceSchema, Value: f012ChildSchemaID, MapTo: "schemaId"}, + }, + }, + }, + } + graph, err := internalworkflow.NewGraph(schema) + require.NoError(t, err) + // Same metadata shape as system.SubWorkflowFunctionMetadata(); normally installed by + // GraphService.EnsureNodeMetadata (workflow_handler.go:139). + require.NoError(t, graph.UpdateNodeMetadata("n2", &packages.FunctionMetadata{ + Input: packages.FunctionInputMetadata{ + Parameters: map[string]workflow.ParameterSchema{ + "schemaId": {Name: "schemaId", Type: "string", Required: true}, + }, + }, + })) + return graph +} + +// f012ChildGraph is a trivial two-node graph standing in for the child schema. +func f012ChildGraph(t *testing.T) *internalworkflow.Graph { + t.Helper() + graph, err := internalworkflow.NewGraph(&internalworkflow.GraphSchema{ + ID: f012ChildSchemaID, + Name: "f01-2 child workflow", + Nodes: []*internalworkflow.NodeSchema{ + {ID: "c1", Function: "debug/nil"}, + {ID: "c2", Function: "debug/nil"}, + }, + Edges: []*internalworkflow.EdgeSchema{ + {ID: "ce1", From: "c1", To: "c2"}, + }, + }) + require.NoError(t, err) + return graph +} + +func f012NewHandler( + wf *internalworkflow.Workflow, + workflowRepo repositories.WorkflowRepository, + journalRepo repositories.JournalRepository, +) (*WorkflowHandler, *f012Process) { + handler := &WorkflowHandler{ + config: &config.Config{}, + workflowRepository: workflowRepo, + journalRepo: journalRepo, + workflow: wf, + forEachStates: make(map[string]*internalworkflow.ForEachState), + iterThreadToForEach: make(map[uint16]string), + } + process := &f012Process{log: &f012Log{}} + handler.Process = process + return handler, process +} + +// f012DistinctChildren collects the DISTINCT child workflow ids that exist for one parent exec id, +// from every observable a child instance can surface in, together with where each id was seen. +// It is deliberately tolerant: an id found in only one source still counts, and message shapes it +// does not recognise are ignored rather than failing, so a fix that changes the internal route +// cannot make this helper report a false duplicate. +func f012DistinctChildren( + t *testing.T, + workflowRepo repositories.WorkflowRepository, + journalRepo repositories.JournalRepository, + parentID string, + parentExecID string, + processes ...*f012Process, +) (childIDs []string, provenance []string) { + t.Helper() + + seen := make(map[string][]string) + + refs, err := workflowRepo.FindActiveSubWorkflows(parentID) + require.NoError(t, err) + for _, ref := range refs { + if ref.ParentExecID.String() != parentExecID { + continue + } + id := ref.ChildWorkflowID.String() + seen[id] = append(seen[id], "subWorkflowRef") + } + + entries, err := journalRepo.LoadAll(parentID) + require.NoError(t, err) + for _, entry := range entries { + if entry.Type != internalworkflow.JournalSubWorkflowStarted || entry.ExecID != parentExecID { + continue + } + if id, ok := entry.Data["childWorkflowId"].(string); ok && id != "" { + seen[id] = append(seen[id], fmt.Sprintf("journal:subworkflow:started#%d", entry.Sequence)) + } + } + + for i, process := range processes { + for _, sent := range process.sent { + message, isMessage := sent.message.(messaging.Message) + if !isMessage || message.Type != messaging.TriggerWorkflow { + continue + } + trigger, isTrigger := message.Args.(messaging.TriggerWorkflowMessage) + if !isTrigger || trigger.SchemaID != f012ChildSchemaID { + continue + } + id := trigger.WorkflowID.String() + seen[id] = append(seen[id], fmt.Sprintf("workflow:trigger(handler#%d)", i+1)) + } + } + + childIDs = make([]string, 0, len(seen)) + for id := range seen { + childIDs = append(childIDs, id) + } + sort.Strings(childIDs) + + provenance = make([]string, 0, len(childIDs)) + for _, id := range childIDs { + provenance = append(provenance, fmt.Sprintf("%s seen in %v", id, seen[id])) + } + return childIDs, provenance +} + +// The invariant under test: one synchronous system/subworkflow step execution owns exactly one +// child workflow instance, across any number of resumes. +func TestWorkflowHandler_ResumeWithPendingSubWorkflow_KeepsExactlyOneDistinctChild(t *testing.T) { + // Arrange — run the parent up to the synchronous sub-workflow node and let the handler + // intercept it once. This is the pre-crash world: one child spawned and in flight, parent + // parked on it, journal persisted. + workflowRepo := repositories.NewMemoryWorkflowRepository() + journalRepo := repositories.NewMemoryJournalRepository() + + wfID := workflow.NewID() + wf := internalworkflow.New(wfID, f012ParentGraph(t), "default") + require.NoError(t, workflowRepo.Save(wf)) + + triggerAction, ok := wf.Trigger().(*workflowactions.RunFunctionAction) + require.True(t, ok, "Trigger must return a RunFunctionAction") + wf.SetResultFor(triggerAction.FunctionExecID, &workflow.FunctionResult{ + Output: workflow.NewFunctionSuccessOutput(map[string]any{}), + }) + subAction, ok := wf.Next(triggerAction.ThreadID).(*workflowactions.RunFunctionAction) + require.True(t, ok, "Next must return a RunFunctionAction for n2") + require.Equal(t, system.SubWorkflowFullFunctionID, subAction.FunctionID) + require.Equal(t, f012ChildSchemaID, subAction.Args["schemaId"]) + parentExecID := subAction.FunctionExecID.String() + + preCrashHandler, preCrashProcess := f012NewHandler(wf, workflowRepo, journalRepo) + preCrashHandler.handleWorkflowAction(subAction) + + preCrashChildren, preCrashProvenance := f012DistinctChildren( + t, workflowRepo, journalRepo, wfID.String(), parentExecID, preCrashProcess) + require.Lenf(t, preCrashChildren, 1, + "precondition: the pre-crash run must own exactly one child (%v)", preCrashProvenance) + require.Equal(t, internalworkflow.StateSleeping, wf.State(), + "precondition: a synchronous sub-workflow parks the parent (workflow_handler.go:951)") + + // The child really is running when the parent dies: WorkflowSupervisor would have spawned a + // handler for it, which Saves the child workflow (workflow_handler.go:172-175). The stub + // process cannot do that, so record it here — otherwise a fix that legitimately respawns only + // when the child never started would be judged against a world where no child exists. + childID := workflow.ID(preCrashChildren[0]) + child := internalworkflow.New(childID, f012ChildGraph(t), "default") + child.SetState(internalworkflow.StateRunning) + require.NoError(t, workflowRepo.Save(child)) + + // Act — restart: rebuild the aggregate from the persisted journal and hand Resume()'s action to + // the handler, exactly as workflow_handler.go:150-159 does on recovery. + entries, err := journalRepo.LoadAll(wfID.String()) + require.NoError(t, err) + require.NotEmpty(t, entries, "the journal must have been persisted before the restart") + + resumed := internalworkflow.New(wfID, f012ParentGraph(t), "default") + resumed.Journal().LoadFrom(entries) + resumeHandler, resumeProcess := f012NewHandler(resumed, workflowRepo, journalRepo) + resumeHandler.handleWorkflowAction(resumed.Resume()) + + // Assert — how many distinct child workflows now exist for that one parent exec. + childIDs, provenance := f012DistinctChildren( + t, workflowRepo, journalRepo, wfID.String(), parentExecID, preCrashProcess, resumeProcess) + assert.Lenf(t, childIDs, 1, + "restart with a pending synchronous system/subworkflow produced %d distinct child workflows for parent exec %s; each of them is a full duplicate execution of schema %q:\n %v", + len(childIDs), parentExecID, f012ChildSchemaID, provenance) +} diff --git a/internal/repositories/graph_memory.go b/internal/repositories/graph_memory.go index 64471c3..879ec8b 100644 --- a/internal/repositories/graph_memory.go +++ b/internal/repositories/graph_memory.go @@ -27,7 +27,13 @@ func NewMemoryGraphRepository() GraphRepository { } } -// FindByID retrieves a graph from the repository +// FindByID retrieves a graph from the repository. +// +// Returns a fresh *workflow.Graph built from a deep-copied schema rather than the stored +// pointer, so a later Save/Upsert for the same id cannot rewrite the topology of a Graph +// already handed out to an in-flight Workflow (BACKLOG_V2.md A-06). This mirrors the +// FindByIDAndVersion idiom below and the Postgres driver, which already rebuilds from the +// object store on every call. func (m *MemoryGraphRepository) FindByID(id string) (*workflow.Graph, error) { m.mu.RLock() defer m.mu.RUnlock() @@ -35,7 +41,8 @@ func (m *MemoryGraphRepository) FindByID(id string) (*workflow.Graph, error) { if !ok { return nil, ErrGraphNotFound } - return graph, nil + schema := graph.Schema() + return workflow.NewGraph(&schema) } // Save stores a graph in the repository diff --git a/internal/services/f01_3_verification_test.go b/internal/services/f01_3_verification_test.go new file mode 100644 index 0000000..fde5acb --- /dev/null +++ b/internal/services/f01_3_verification_test.go @@ -0,0 +1,200 @@ +package services_test + +// BACKLOG_V2.md F-01 finding 3 (BACKLOG_V2.md:53) verification — memory driver half. +// +// Premise under test (verbatim): "graph_memory.go:FindByID returns the shared m.graphs[id] pointer +// live Workflow's hold, so updateVersioned -> graph.UpdateSchema rewrites a running run's topology +// in place." +// +// WHAT THIS PROVES: with the in-memory GraphRepository wired into DefaultGraphService, a run built +// the way internal/actors/workflow_handler.go:167-177 builds it (graphService.FindByID -> +// internalworkflow.New -> Trigger) has its topology rewritten underneath it by a later +// GraphService.Upsert of a different schema for the same id — it loses a node, loses a parallel +// branch at the next scheduling decision, and panics in Workflow.Next when the node it is executing +// is the one that disappeared. +// +// WHAT THIS DOES NOT PROVE: nothing about pointer identity or copy semantics anywhere — no test +// here asserts that any two values are or are not the same pointer, because that is the shape of a +// fix, not the shape of the defect. Nothing about the Postgres driver (see +// tests/functional/f01_3_verification_test.go for the parity contract), nothing about actor +// scheduling, and nothing about process restart. No ergo node is started here; the run is driven +// through the workflow aggregate directly, which is the same object the actor holds. + +import ( + "testing" + + "github.com/open-source-cloud/fuse/internal/mocks" + "github.com/open-source-cloud/fuse/internal/packages" + "github.com/open-source-cloud/fuse/internal/repositories" + "github.com/open-source-cloud/fuse/internal/services" + internalworkflow "github.com/open-source-cloud/fuse/internal/workflow" + "github.com/open-source-cloud/fuse/internal/workflow/workflowactions" + "github.com/open-source-cloud/fuse/pkg/llm" + pkgworkflow "github.com/open-source-cloud/fuse/pkg/workflow" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// newF013GraphService builds a GraphService backed by the in-memory graph repository with the +// internal packages registered, so node metadata population succeeds. +func newF013GraphService(t *testing.T) services.GraphService { + t.Helper() + + memGraphRepo := repositories.NewMemoryGraphRepository() + pkgRegistry := packages.NewPackageRegistry() + internalPackages := packages.NewInternal(llm.NewRegistry(nil, ""), pkgRegistry, nil) + pkgSvc := services.NewPackageService(repositories.NewMemoryPackageRepository(), pkgRegistry, internalPackages) + require.NoError(t, pkgSvc.RegisterInternalPackages()) + + return services.NewGraphService(memGraphRepo, pkgRegistry, nil) +} + +// Node ids inside mocks.SmallTestGraphSchema that this file manipulates. +const ( + f013TriggerNode = "debug-nil" + f013BranchNode = "logic-rand-2" + f013RenamedTriggerNode = "debug-nil-renamed" +) + +// f013LinearSchema returns the same schema id as mocks.SmallTestGraphSchema but with the +// logic-rand-2 branch removed, so debug-nil forks into 2 nodes in v1 and into 1 in v2. +func f013LinearSchema() *internalworkflow.GraphSchema { + schema := mocks.SmallTestGraphSchema() + + nodes := make([]*internalworkflow.NodeSchema, 0, len(schema.Nodes)) + for _, n := range schema.Nodes { + if n.ID == f013BranchNode { + continue + } + nodes = append(nodes, n) + } + schema.Nodes = nodes + + edges := make([]*internalworkflow.EdgeSchema, 0, len(schema.Edges)) + for _, e := range schema.Edges { + if e.From == f013BranchNode || e.To == f013BranchNode { + continue + } + edges = append(edges, e) + } + schema.Edges = edges + + return schema +} + +// TestGraphService_Upsert_MustNotMutateRunningWorkflowGraph asserts the specific corruption: a +// workflow that already triggered on schema v1 observes v2's node set without being restarted. +func TestGraphService_Upsert_MustNotMutateRunningWorkflowGraph(t *testing.T) { + // Arrange + svc := newF013GraphService(t) + v1 := mocks.SmallTestGraphSchema() + _, err := svc.Upsert(v1.ID, v1) + require.NoError(t, err) + + // Mirrors internal/actors/workflow_handler.go:167-177 on the create path. + graphRef, err := svc.FindByID(v1.ID) + require.NoError(t, err) + run := internalworkflow.New(pkgworkflow.NewID(), graphRef, "default") + require.NotNil(t, run.Trigger(), "precondition: the run started") + + triggerNode, err := run.Graph().FindNode(f013TriggerNode) + require.NoError(t, err) + require.Len(t, triggerNode.OutputEdges(), 2, "precondition: the run started on the forking v1 topology") + _, err = run.Graph().FindNode(f013BranchNode) + require.NoError(t, err, "precondition: logic-rand-2 exists in v1") + + // Act — a schema edit arrives through the ordinary public path while the run is in flight. + _, err = svc.Upsert(v1.ID, f013LinearSchema()) + require.NoError(t, err) + + // Assert + _, err = run.Graph().FindNode(f013BranchNode) + assert.NoError(t, err, + "the in-flight run's graph lost node logic-rand-2: updateVersioned -> Graph.UpdateSchema rewrote the running run's topology in place") + + triggerNodeAfter, err := run.Graph().FindNode(f013TriggerNode) + require.NoError(t, err) + assert.Len(t, triggerNodeAfter.OutputEdges(), 2, + "the in-flight run's fork degraded from 2 output edges to %d after an unrelated schema upsert", len(triggerNodeAfter.OutputEdges())) +} + +// f013RenamedTriggerSchema returns the same schema id with the trigger node renamed, so a run that +// is in flight on "debug-nil" finds no such node after the edit. +func f013RenamedTriggerSchema() *internalworkflow.GraphSchema { + schema := f013LinearSchema() + for _, n := range schema.Nodes { + if n.ID == f013TriggerNode { + n.ID = f013RenamedTriggerNode + } + } + for _, e := range schema.Edges { + if e.From == f013TriggerNode { + e.From = f013RenamedTriggerNode + } + if e.To == f013TriggerNode { + e.To = f013RenamedTriggerNode + } + } + return schema +} + +// TestWorkflow_Next_AfterLiveSchemaEditDropsInFlightNode_MustNotPanic asserts the worst consequence of +// the shared pointer: Workflow.Next ignores the error from graph.FindNode +// (internal/workflow/workflow.go:312) and dereferences the nil node on the next line. +func TestWorkflow_Next_AfterLiveSchemaEditDropsInFlightNode_MustNotPanic(t *testing.T) { + // Arrange + svc := newF013GraphService(t) + v1 := mocks.SmallTestGraphSchema() + _, err := svc.Upsert(v1.ID, v1) + require.NoError(t, err) + + graphRef, err := svc.FindByID(v1.ID) + require.NoError(t, err) + run := internalworkflow.New(pkgworkflow.NewID(), graphRef, "default") + trigger, ok := run.Trigger().(*workflowactions.RunFunctionAction) + require.True(t, ok) + + // Act + _, err = svc.Upsert(v1.ID, f013RenamedTriggerSchema()) + require.NoError(t, err) + + // Assert + assert.NotPanics(t, func() { _ = run.Next(trigger.ThreadID) }, + "the node the run is executing was deleted from its graph in place, and Next dereferences the nil lookup result") +} + +// TestWorkflow_Next_AfterLiveSchemaEdit_MustKeepParallelBranch asserts the behavioural consequence: +// the same Next() call that yields a 2-way parallel action on an isolated v1 graph yields a +// single-function action on the run whose graph the repository handed out by pointer. +func TestWorkflow_Next_AfterLiveSchemaEdit_MustKeepParallelBranch(t *testing.T) { + // Arrange + svc := newF013GraphService(t) + v1 := mocks.SmallTestGraphSchema() + _, err := svc.Upsert(v1.ID, v1) + require.NoError(t, err) + + victimGraph, err := svc.FindByID(v1.ID) + require.NoError(t, err) + victim := internalworkflow.New(pkgworkflow.NewID(), victimGraph, "default") + victimTrigger, ok := victim.Trigger().(*workflowactions.RunFunctionAction) + require.True(t, ok) + + controlGraph, err := internalworkflow.NewGraph(mocks.SmallTestGraphSchema()) + require.NoError(t, err) + control := internalworkflow.New(pkgworkflow.NewID(), controlGraph, "default") + controlTrigger, ok := control.Trigger().(*workflowactions.RunFunctionAction) + require.True(t, ok) + + // Act — upsert the linear v2 schema; only the repository-held graph should be affected. + _, err = svc.Upsert(v1.ID, f013LinearSchema()) + require.NoError(t, err) + + controlNext := control.Next(controlTrigger.ThreadID) + victimNext := victim.Next(victimTrigger.ThreadID) + + // Assert + require.Equal(t, workflowactions.ActionRunParallelFunctions, controlNext.Type(), + "control run (graph never handed out by the repository) still forks") + assert.Equal(t, workflowactions.ActionRunParallelFunctions, victimNext.Type(), + "in-flight run scheduled %s instead of a 2-way parallel fork after a concurrent schema upsert", victimNext.Type()) +} diff --git a/tests/functional/f01_3_verification_pg_test.go b/tests/functional/f01_3_verification_pg_test.go new file mode 100644 index 0000000..6bf7e6f --- /dev/null +++ b/tests/functional/f01_3_verification_pg_test.go @@ -0,0 +1,28 @@ +//go:build functional + +package functional_test + +// Postgres invocation of the F-01.3 parity contract. The shared body, and what it does and does not +// prove, is documented in f01_3_verification_test.go. Skips silently without DB_POSTGRES_DSN +// (tests/functional/postgres_test.go:25), so a green run here proves nothing unless the DSN was set. + +import ( + "context" + "testing" + + "github.com/open-source-cloud/fuse/internal/repositories" + "github.com/open-source-cloud/fuse/internal/repositories/postgres" + "github.com/stretchr/testify/require" +) + +func TestPostgresGraphRepository_LiveSchemaEditIsolation_Contract(t *testing.T) { + pool := setupTestPool(t) + store := testObjectStore() + contractTestGraphLiveSchemaEditIsolation(t, func() repositories.GraphRepository { + return postgres.NewGraphRepository(pool, store) + }, func() { + _, err := pool.Exec(context.Background(), + "TRUNCATE TABLE graph_schema_versions, graph_schema_nodes, graph_schema_metadata, graph_schema_tags, graph_schemas CASCADE") + require.NoError(t, err) + }) +} diff --git a/tests/functional/f01_3_verification_test.go b/tests/functional/f01_3_verification_test.go new file mode 100644 index 0000000..920056c --- /dev/null +++ b/tests/functional/f01_3_verification_test.go @@ -0,0 +1,129 @@ +package functional_test + +// BACKLOG_V2.md F-01 finding 3 (BACKLOG_V2.md:53) — driver-parity half. +// +// Premise under test (verbatim): "graph_memory.go:FindByID returns the shared m.graphs[id] pointer +// live Workflow's hold, so updateVersioned -> graph.UpdateSchema rewrites a running run's topology +// in place." +// +// WHAT THIS PROVES: for each driver, whether a workflow run that is already in flight on schema v1 +// keeps executing v1's topology after an ordinary schema edit (GraphService.Upsert) lands for the +// same schema id. The run is built exactly the way internal/actors/workflow_handler.go:167-177 +// builds it — graphService.FindByID -> internalworkflow.New -> Trigger — and the edit arrives +// through the ordinary public path, so nothing here presumes where the corruption is introduced or +// where a fix would go. +// +// WHAT THIS DOES NOT PROVE: nothing about pointer identity, aliasing or copy semantics inside any +// repository — those are implementation choices, deliberately not asserted. Nothing about actor +// scheduling, process restart, or the durable half of the same problem (A-05). No ergo node is +// started; the run is driven through the workflow aggregate, which is the object the actor holds. +// +// The memory invocation below is untagged and runs under `make test`; the Postgres invocation lives +// in f01_3_verification_pg_test.go behind //go:build functional and needs DB_POSTGRES_DSN +// (tests/functional/postgres_test.go:25 skips silently without it). + +import ( + "testing" + + "github.com/open-source-cloud/fuse/internal/mocks" + "github.com/open-source-cloud/fuse/internal/packages" + "github.com/open-source-cloud/fuse/internal/repositories" + "github.com/open-source-cloud/fuse/internal/services" + internalworkflow "github.com/open-source-cloud/fuse/internal/workflow" + "github.com/open-source-cloud/fuse/pkg/llm" + pkgworkflow "github.com/open-source-cloud/fuse/pkg/workflow" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// Node ids inside mocks.SmallTestGraphSchema that this file manipulates. +const ( + f013TriggerNode = "debug-nil" + f013BranchNode = "logic-rand-2" +) + +// f013LinearSchema returns mocks.SmallTestGraphSchema with the logic-rand-2 branch removed, so the +// two schemas share an ID but differ in topology: debug-nil forks into 2 nodes in v1 and into 1 in +// v2. +func f013LinearSchema() *internalworkflow.GraphSchema { + schema := mocks.SmallTestGraphSchema() + + nodes := make([]*internalworkflow.NodeSchema, 0, len(schema.Nodes)) + for _, n := range schema.Nodes { + if n.ID == f013BranchNode { + continue + } + nodes = append(nodes, n) + } + schema.Nodes = nodes + + edges := make([]*internalworkflow.EdgeSchema, 0, len(schema.Edges)) + for _, e := range schema.Edges { + if e.From == f013BranchNode || e.To == f013BranchNode { + continue + } + edges = append(edges, e) + } + schema.Edges = edges + + return schema +} + +// f013GraphService builds a GraphService over the driver's repository with the internal packages +// registered, matching how the server wires it (internal/app/di). +func f013GraphService(t *testing.T, repo repositories.GraphRepository) services.GraphService { + t.Helper() + + pkgRegistry := packages.NewPackageRegistry() + internalPackages := packages.NewInternal(llm.NewRegistry(nil, ""), pkgRegistry, nil) + pkgSvc := services.NewPackageService(repositories.NewMemoryPackageRepository(), pkgRegistry, internalPackages) + require.NoError(t, pkgSvc.RegisterInternalPackages()) + + return services.NewGraphService(repo, pkgRegistry, nil) +} + +// contractTestGraphLiveSchemaEditIsolation is the shared body both drivers are held to: a run that +// is already in flight must not observe a schema edit that lands after it started. +func contractTestGraphLiveSchemaEditIsolation(t *testing.T, newRepo func() repositories.GraphRepository, reset func()) { + t.Helper() + + t.Run("an in-flight run keeps the topology it started on across a schema edit", func(t *testing.T) { + // Arrange — schema v1 exists and a run is started on it. + reset() + svc := f013GraphService(t, newRepo()) + v1 := mocks.SmallTestGraphSchema() + _, err := svc.Upsert(v1.ID, v1) + require.NoError(t, err) + + graphRef, err := svc.FindByID(v1.ID) + require.NoError(t, err) + run := internalworkflow.New(pkgworkflow.NewID(), graphRef, "default") + require.NotNil(t, run.Trigger(), "precondition: the run started") + + triggerNode, err := run.Graph().FindNode(f013TriggerNode) + require.NoError(t, err) + require.Len(t, triggerNode.OutputEdges(), 2, "precondition: the run started on the forking v1 topology") + _, err = run.Graph().FindNode(f013BranchNode) + require.NoError(t, err, "precondition: logic-rand-2 is part of v1") + + // Act — an ordinary schema edit for the same id lands while the run is in flight. + _, err = svc.Upsert(v1.ID, f013LinearSchema()) + require.NoError(t, err) + + // Assert — the in-flight run still sees v1. + _, err = run.Graph().FindNode(f013BranchNode) + assert.NoError(t, err, + "the in-flight run lost node logic-rand-2: a schema edit rewrote the topology of a run that had already started") + + triggerNodeAfter, err := run.Graph().FindNode(f013TriggerNode) + if assert.NoError(t, err, "the in-flight run lost its trigger node debug-nil after a schema edit") { + assert.Len(t, triggerNodeAfter.OutputEdges(), 2, + "the in-flight run's fork degraded from 2 output edges to %d after a schema edit it should not see", + len(triggerNodeAfter.OutputEdges())) + } + }) +} + +func TestMemoryGraphRepository_LiveSchemaEditIsolation_Contract(t *testing.T) { + contractTestGraphLiveSchemaEditIsolation(t, repositories.NewMemoryGraphRepository, func() {}) +}