diff --git a/doc/rfc/hook-framework.md b/doc/rfc/hook-framework.md new file mode 100644 index 00000000..77305ae5 --- /dev/null +++ b/doc/rfc/hook-framework.md @@ -0,0 +1,123 @@ +# Hooks Framework + +Fire-and-forget side effects for pipeline lifecycle events: one shared event contract, a durable hook topic per domain, pluggable hooks. + +## Problem + +The pipelines emit lifecycle transitions — a request lands or fails, a batch merges, a build finishes — but nothing can react outside pipeline state: no warehouse export, no PR comments or closes on merge events, no notifications or audit trails. The log topic is not this seam: SubmitQueue request statuses only, consumed solely to build gateway read models. + +Two requirements: side effects must never stall or fail the pipeline, and "fire and forget" must not mean lossy — a merge-failure comment that silently never posts is a support ticket. + +## Proposal + +When a controller performs a transition, it also publishes a **hook event** to a durable `hook` topic. A thin per-domain dispatcher stage consumes it and hands each event to the **hooks** the host wired — no-op by default, real integrations as they arrive. + +``` +pipeline controller dispatcher stage (per domain) +state write → hook publish → downstream ──▶ [hook topic] ──▶ decode → validate → hook.Handle + │ ├─ noop (default) + │ retries exhausted └─ composite ─▶ warehouse, code host, … + ▼ + [hook_dlq] ──▶ log full event + page; manual republish +``` + +One event shape for every domain — the CloudEvents core plus a version ordinal: + +| Field | What it is | Why it is on the envelope | +|---|---|---| +| `id` | Opaque occurrence identity, publisher-minted (`source/type/subject-id/version`) | Queue dedupe key and hook idempotency key | +| `source` | Producing domain (`submitqueue`) | Keeps multi-domain sinks unambiguous | +| `type` | What happened, one dotted open string (`batch.failed`) | The single filter dimension | +| `timestamp_ms` | Occurrence time, ms since epoch | Uniform time axis | +| `version` | Subject's ordinal at the transition; 0 = not a versioned write | Staleness detection under at-least-once — wall clocks can't | +| `payload` | Per-type facts, always including the subject's id | Everything else; open, additive vocabulary | + +Delivery promise: + +- At least once, deduped by `id` within the queue's retention window; hooks are idempotent by `id`. +- The hook publish rides the delivery that causes the transition — state write, hook publish, downstream publishes, ack — so a crash replays everything. No outbox. +- A failed publish fails the stage (the hook topic shares the pipeline's queue backend); loss is never silent. +- Past the retry budget, the DLQ logs the complete event and pages; manual republish recovers it. +- Ordering per subject only. Hook outcomes never write pipeline state. + +## Decisions + +### Identity and replay + +- `id` = source / type / subject id / post-transition version; the causal message id stands in when unversioned, plus an ordinal for multiple same-typed events per cause. Components are separator-free. Consumers never parse it. +- Partition key = subject id; per-subject order only. +- Replay finding the target state already written → republish (idempotent). Beyond it → superseded; the event may be lost. Named, accepted gap. +- RPC-caused transitions are at-most-once; needing the guarantee means publishing from the first queue-driven stage. +- Opt-in per host via topic-key registration; a registered host never skips, so off and loss are distinguishable. + +### Contract + +- `api/base/hook/`: no owning domain, so the message-queue location rule extends — platform-owned contracts live under `api/base/`. +- Envelope = only fields every consumer keys on uniformly; subject, queue, and error are occurrence facts → payload. `source`/`type` are strings, not enums, for additive evolution. +- Payload (`Struct`): shaped per type, add-only, documented by its domain; must carry the subject's id and transient facts (merge step outcomes, build failure detail) — the event is their only durable record. Never entity snapshots; hooks resolve entities from stores. + +### Hooks and dispatch + +- Extension at `platform/extension/hook/`, singleton shape (counter precedent), wired once per host; no per-queue factory. +- Hook contract: at-least-once, idempotent by `id`, plain errors, never writes pipeline state; ignore an event by returning nil (no filter API). +- Ships `noop` (default) and `composite` (runs all children, joins failures, names failing children). A cross-domain sink is the same impl wired into each domain. +- Dispatcher: decode, validate (`id`/`source`/`type` non-empty), invoke. Malformed events dead-letter, never silently acked; hook errors retry then dead-letter, with errs classifiers fast-pathing permanent failures. +- DLQ reconciler: log the full event with its failure attribution, page (new metric — the log DLQ only warns), then ack. Manual republish recovers; pipeline state is never touched. +- Per-hook retry isolation later: consumer groups on the same `hook` topic key, once the registry supports multiple groups per key and rejection becomes group-local (today it moves the shared row). Until then the composite's shared budget is accepted. + +## Example + +```proto +syntax = "proto3"; + +package uber.base.hook; + +import "google/protobuf/struct.proto"; + +import "api/base/messagequeue/proto/messagequeue.proto"; + +// HookEvent is one fire-and-forget lifecycle event. Every domain publishes +// this same shape to its own hook topic; hook implementations consume it. +message HookEvent { + option (uber.base.messagequeue.topic_keys) = "hook"; + + string id = 1; // Opaque occurrence identity; queue message id and hook idempotency key. + string source = 2; // Producing domain: "submitqueue", "stovepipe", ... + string type = 3; // What happened: "request.landed", "batch.failed", ... + int64 timestamp_ms = 4; // Occurrence time, ms since the Unix epoch. + int32 version = 5; // Subject's version at the transition; 0 when not tied to a state write. + google.protobuf.Struct payload = 6; // Publisher-defined facts, including the subject's id; never a snapshot. +} +``` + +A failed batch, carrying merge-result facts persisted nowhere else (protojson: int64 as string, empty fields omitted): + +```json +{ + "id": "submitqueue/batch.failed/batch-778/4", + "source": "submitqueue", + "type": "batch.failed", + "timestamp_ms": "1722800012345", + "version": 4, + "payload": { + "batch_id": "batch-778", + "queue": "go-monorepo", + "error": "merge conflict", + "failed_step": "sq-12346", + "conflict_paths": ["foo/bar.go"] + } +} +``` + +## Rejected + +- **A contract per domain.** N schemas, N hook shapes, N warehouse tables; one envelope absorbs differences additively. +- **Inline hook calls.** Couples pipeline latency to integrations; a crash between write and call silently drops the notification. +- **A second consumer group on the log topic.** Request statuses only; no path to batch, build, merge, or other domains. +- **Enums for source/type.** protojson rejects unknown enum values; every addition would break consumers. +- **Subject, queue, or error on the envelope.** Occurrence facts; they live in the payload. No major event platform carries a top-level error. +- **Entity snapshots as payload.** Stale on redelivery; competes with the store; drags domain schemas into the shared contract. +- **Typed per-event payloads (`oneof`).** Every new type becomes a wire-contract change. +- **A filter/subscription API.** Returning nil costs nothing; routing can be a wiring decorator later. +- **Best-effort publishing.** Silent loss; failing the stage is safe because dedupe makes the retry idempotent. +- **A transactional outbox.** The publish rides the triggering delivery before ack; a crash replays both. RPC-edge transitions are scoped out instead. diff --git a/doc/rfc/index.md b/doc/rfc/index.md index 37e0d867..d44b9860 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -9,6 +9,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting - [Consumer Gate](consumer-gate.md) - Stopping and starting individual queue controllers at runtime via consumer middleware: parked deliveries held in-flight with visibility extension, gate state as a separate extension with a file-based first implementation shared by tests and operators - [Consumer Hold](consumer-hold.md) - Fourth delivery outcome letting a controller postpone its delivery: the message becomes a partition barrier that pauses consumption for a chosen delay, redelivers in order, and does not count as a failure toward dead-lettering - [Change URIs](change-uri.md) - Identity of a code change: `scheme://{host[:port]}/{path}` per provider (GitHub PR, Phabricator Diff, git ref/commit) and canonical-form rules +- [Hooks Framework](hook-framework.md) - Fire-and-forget side effects off pipeline lifecycle events: one shared `HookEvent` contract (`api/base/hook/`) published to a durable per-domain hook topic, dispatched by a per-domain stage to a pluggable hook extension (`platform/extension/hook/`) for integrations like warehouse export and code-review notifications ## SubmitQueue