feat: deploy-governance suite — change analysis, backfill, weekly summaries

deploy-governance/README.md

@@ -0,0 +1,177 @@
+# deploy-governance
+
+Deploy governance without a human bottleneck: every production deployment
+gets an **automatic change analysis** (what changed, who touched it, how risky
+it is), the result gates the deploy through a **risk × criticality matrix**,
+and the accumulated data rolls up into **weekly functional summaries** per
+application and namespace — release notes nobody had to write.
+
+This is the suite nullplatform runs on its own organization (dogfood). It
+replaced "someone self-approves in Slack" with evidence-based gates.
+
+## The system
+
+```
+deploy to production
+        │
+        ▼ (checklist external item, kind: deploy-change-analysis)
+┌─────────────────────────┐    per-deploy analysis:
+│ deploy-change-analysis  │──► diff deployed→candidate release (GitHub compare)
+│ (np-checklist-trigger)  │    PRs as rich objects (author, collaborators+roles,
+└───────────┬─────────────┘    description, LLM one-line summary)
+            │                  deterministic risk signals (db migrations, auth,
+            │                  infra, deps paths) + LLM score bounded by floors
+            │                  light matrix: risk × app criticality → auto/peer/group
+            ▼
+  deployment.metadata.change   (the durable artifact everything else reads)
+            ▲
+┌───────────┴─────────────┐
+│ deploy-backfill         │  nightly cron (+ manual webhook): finds prod deploys
+│                         │  WITHOUT metadata in the last N days and fires the
+└─────────────────────────┘  analysis for each — self-healing safety net
+            │
+            ▼ (weekly cron, well after the nightly)
+┌─────────────────────────┐  ONE LLM call aggregates the week's already-written
+│ deploy-weekly-summary   │  per-deploy summaries into app summaries + namespace
+└───────────┬─────────────┘  roll-ups. No GitHub calls, no re-analysis.
+            ▼
+  application/deploy_summaries + namespace/deploy_summaries
+  (rolling 53 weeks; the catalog card shows latest_summary only)
+```
+
+## Workflows
+
+| File | Trigger | Purpose |
+|---|---|---|
+| `deploy-change-analysis.yaml` | `np-checklist-trigger`, kind `deploy-change-analysis` | Resolves the checklist external item on real deploys: analysis → metadata → item resolution with markdown + per-gate results |
+| `deploy-change-analysis-manual.yaml` | webhook | Same pipeline behind a plain webhook: smoke tests and backfill children. Payload mirrors the trigger outputs (`runId`, `itemId`, `callbackUrl`, `callbackToken`, `inputs.*`) plus backfill overrides (`from_release_id`, `previously_deployed`) |
+| `deploy-backfill.yaml` | cron (nightly) + webhook (manual) | Lake query for prod deploys missing `metadata.change` → fires the manual webhook per deploy. Skip-existing makes it idempotent and convergent |
+| `deploy-weekly-summary.yaml` | cron (weekly) | Aggregates the week per app/namespace, one LLM call, writes rolling summaries |
+
+`templates/checklist-template-cross-validation.yaml` is the target checklist
+template: the analysis item (informational) plus a `cross_validation` group
+(`aggregation: any`) where already-deployed fastpath, the risk matrix, a
+four-eyes pair review, and an admin escalation each unblock the deploy.
+
+## Setup
+
+1. **Catalog specs** (one-time):
+   ```bash
+   NP_API_KEY=... NP_ORGANIZATION_ID=<org> ./setup/01-catalog-specs.sh
+   ```
+   Then classify applications (`governance.criticality`, editable in the UI)
+   and map users (`identity.github_username`).
+
+2. **Config entries** (never in YAML):
+
+   | Entry | Kind | Scope | Used by |
+   |---|---|---|---|
+   | `NP_API_KEY` | secret | each workflow | NP API + lake access |
+   | `GITHUB_TOKEN` | secret | analysis workflows | compare/PR/review reads |
+   | `NP_ORGANIZATION_ID` | var | analysis workflows | trigger NRN + user listing |
+   | `CHANGE_ANALYSIS_WEBHOOK` | secret | backfill | activated webhook URL of the manual variant (token-bearing → secret) |
+
+   Note: a workflow's `path:` does **not** inherit folder config — scope
+   entries to each workflow.
+
+3. **Publish** (`npx np-workflow publish <file> --alias live`), activate, then
+   set `CHANGE_ANALYSIS_WEBHOOK` to the URL minted for the manual variant
+   (`GET /workflows/triggers?workflowId=...`).
+
+4. **Backfill history**: `POST` the backfill webhook with
+   `{"days": 14, "application_id": 0}` (0 = all apps). Re-fire until it
+   reports `fired: 0` — see gotchas.
+
+## Metadata contract: `deployment.metadata.change`
+
+The durable artifact of the suite. Its specification carries a **full JSON
+Schema** (`specs/deployment-change.spec.json`, upserted by the setup script)
+so the dashboard renders it schema-driven instead of dumping a JSON blob.
+
+What the analysis writes per deploy (all levels `additionalProperties: true`):
+
+| Field | Shape | Notes |
+|---|---|---|
+| `risk` | `low \| medium \| high` | LLM score bounded by deterministic floors |
+| `short_summary` / `summary_md` | string / markdown | one-liner + full narrative (PRs, participants, rationale) |
+| `risk_rationale`, `risk_floors_applied` | string, string[] | floors: `db_migration`, `auth_change`, `first_deploy` |
+| `change_categories` | `[{category, count, notes}]` | LLM taxonomy (feature/bugfix/security/…); `category` is deliberately a free string in the schema so historical docs never fail write-validation |
+| `breaking`, `hotfix` | boolean | only present when true |
+| `approval` | `{mode, criticality}` | matrix decision: `auto \| fastpath \| par \| grupo` |
+| `from_release` / `to_release` | `{id, semver, commit_sha}` | `from_release` is `null` on first deploys |
+| `releases_between`, `previously_deployed` | int, bool | accumulation + rollback fastpath |
+| `signals` | sizes + `sensitive_paths{}` | deterministic inputs to the risk score |
+| `prs` | rich PR objects | number/title/author/summary/size + per-PR `ai {used, level}` |
+| `participants` | `[{github, np_user_id?, roles}]` | `np_user_id` null until mapped via `user/identity` |
+| `ai_usage` | `{prs_ai, prs_total, commits_ai, commits_total}` | declared-AI lower bound |
+
+**Visibility contract**: the schema declares `visibleOn: ["read"]` at the
+root — the document renders on the **deployment detail** only. It never
+becomes deployment-list columns (nothing is marked `visibleOn: list`, which
+list columns require per property) and never appears in create/update forms
+(it is machine-written). Raw payloads (`commits`, `files`, long PR
+descriptions, plumbing ids) are intentionally **not declared** as properties:
+`additionalProperties: true` keeps accepting them on writes, but the
+schema-driven UI does not render them — `summary_md` already narrates that
+content.
+
+**Evolving the schema**: the metadata service validates every write against
+it, so a stricter schema can brick the analysis pipeline. Before changing
+`specs/deployment-change.spec.json`, validate a sample of real stored docs
+against the new schema (AJV 8, `strict: false` — same as the service), then
+re-run `setup/01-catalog-specs.sh` (it PATCHes the existing spec in place).
+
+## Cron layout
+
+- Backfill: nightly (e.g. `30 1 * * *`), window 3 days — self-heals gaps.
+- Weekly summary: e.g. Mondays `0 9 * * 1` — hours AFTER the nightly, so the
+  closing week is fully analyzed before it is summarized.
+
+## Production gotchas this suite encodes
+
+- **Trigger config is literal**: `${{ vars.* }}` does NOT resolve inside a
+  trigger's `config` at activation (steps resolve at run time; triggers do
+  not) — the `np-checklist-trigger` `nrn` must be a literal, or channel
+  creation fails with an opaque 401. Activation also needs a caller with
+  `notification_channel` permissions (the activate call's bearer is the
+  actor for channel management).
+
+- **Sandbox pool saturation**: firing 100+ analysis children at once exhausts
+  the code-exec sandbox pool; children die *before* the LLM step (zero cost).
+  The dispatcher skips deploys that already have metadata, so re-firing
+  converges instead of re-paying. Failures show as `SANDBOX_NOT_AVAILABLE`
+  or as executions that "completed" through their failure-resolve fallback —
+  check step statuses, not execution status.
+- **GitHub reads are parallel**: per-commit PR lookups and per-PR reviews run
+  in chunks of 10. Sequential, a 100-commit diff exceeds the sandbox budget.
+- **First deploys**: the lake's TSV `NULL` (`\N`) for `lag()` means "first
+  deploy of this scope". The dispatcher sends the explicit `"none"` sentinel;
+  the analysis then keeps `from = null` instead of wrongly diffing against
+  *today's* current release.
+- **Agent prompt size**: the weekly summary sends a compact `llm_view`
+  (summaries + PR titles only) — the full gathered object on a busy week
+  (~180KB) crashes the agent runner. Keep agent prompts under ~100KB.
+- **Agent tool detours**: data-in → structured-out agent steps need
+  "do NOT use tools, respond directly" in the system prompt and enough
+  `maxIterations` headroom (15), or the model burns its turns exploring.
+- **Metadata writes are schema-validated** (with AJV type coercion, so
+  mismatches produce confusing errors like `prs/0 must be object`). Each
+  spec must declare exactly what its workflow writes.
+- **Catalog UI keys** live *inside* schema properties: `visibleOn`
+  (`create|read|update|list`; `[]` hides a field from the UI while keeping
+  it via API), `uiSchema`, `tag`. The dashboard card renders the `read`
+  context — this suite shows `latest_summary` only and keeps `weeks` as
+  API/lake data.
+- **Lake over HTTP**: `POST /data/lake/query {query}` returns *headerless*
+  TSV — parse positionally, expect numbers as strings, and add `FINAL` to
+  versioned tables or rows duplicate.
+
+## Tests
+
+The suites' usual plugin-level stubbing does not apply here: this suite's
+I/O happens inside `code-exec` sandboxes (raw `fetch` to the NP API, the
+lake and GitHub), not through stubbable integration plugins. Validate with
+`npx np-workflow validate <file>` (all four pass, including the dual graph
+pass) — behavioral coverage comes from the manual-webhook variant, which
+runs the full pipeline against a real deployment without touching any
+checklist run.

deploy-governance/deploy-backfill.yaml

@@ -0,0 +1,168 @@
+# Deploy Backfill — computes deployment.metadata.change for historical
+# production deploys by firing the deploy-change-analysis-manual webhook
+# once per deploy. Single code-exec step: lake query + payload build +
+# webhook fan-out (children run as independent engine executions).
+#
+# Fire manually:
+#   POST <webhook>/checklist/deploy/backfill
+#   body: {"days": 15, "application_id": <app-id>}   # application_id: 0 = all apps
+#
+# NOTE: this engine does not resolve `${{ steps.X.outputs.* }}` inside step
+# `inputs:` (arrives empty in the sandbox) — hence the single-step design.
+id: deploy-backfill
+key: deploy-backfill
+name: "Deploy Backfill (change analysis)"
+description: >
+  Queries the lake for finalized production deployments in the last N days,
+  computes each deploy's previous release at that point in time, and fires
+  the change-analysis workflow webhook per deploy with from_release_id /
+  previously_deployed overrides.
+semantic_version: 1.2.0
+path: "/checklist/deploy"
+
+inputs:
+  triggerPayload:
+    type: object
+    default: {}
+
+steps:
+  - id: trigger
+    type: trigger
+    pluginType: webhook
+    name: "Manual backfill trigger"
+    config:
+      path: /checklist/deploy/backfill
+      method: POST
+      mode: start
+
+  # Nightly safety net until the checklist gates every prod deploy: catches
+  # any production deployment that finished without change metadata (deploys
+  # while the pilot action is detached, workflow hiccups, etc.). The 3-day
+  # window + skip-existing makes it self-healing and idempotent.
+  - id: trigger_nightly
+    type: trigger
+    pluginType: cron
+    name: "Nightly backfill (01:30 AR, last 3 days)"
+    config:
+      schedule: "30 1 * * *"
+      timezone: "America/Argentina/Buenos_Aires"
+
+  - id: run
+    type: module
+    pluginType: code-exec
+    name: "Lake pairs + fan-out webhooks"
+    inputs:
+      days: "${{ workflow.inputs.body.days }}"
+      application_id: "${{ workflow.inputs.body.application_id }}"
+      np_api_key: "${{ secrets.NP_API_KEY }}"
+      child_webhook: "${{ secrets.CHANGE_ANALYSIS_WEBHOOK }}"
+    config:
+      network:
+        allowedHosts: ["api.nullplatform.com"]
+      code: |
+        // Default 3: the nightly cron fires with no body — a 3-day window
+        // self-heals weekend gaps. Manual webhook calls pass days explicitly.
+        const days = Number($item.days) || 3;
+        const appId = Number($item.application_id) || 0;
+        const npApiKey = $item.np_api_key;
+        // Activated webhook URL of deploy-change-analysis-manual (token-bearing,
+        // minted at alias activation) — configured as a workflow var, never in YAML.
+        const CHILD_WEBHOOK = $item.child_webhook;
+        if (!CHILD_WEBHOOK) return { stage: "config", error: "CHANGE_ANALYSIS_WEBHOOK var is not set" };
+
+        let stage = "token";
+        const tokRes = await fetch("https://api.nullplatform.com/token", {
+          method: "POST",
+          headers: { "content-type": "application/json" },
+          body: JSON.stringify({ api_key: npApiKey }),
+        });
+        const tokText = await tokRes.text();
+        let npToken;
+        try { npToken = JSON.parse(tokText).access_token; }
+        catch (e) { return { stage, error: String(e), sample: tokText.slice(0, 200) }; }
+        if (!npToken) return { stage, error: "no access_token", sample: tokText.slice(0, 200) };
+
+        const sql = `
+          WITH prod_scopes AS (
+            SELECT DISTINCT scope_id FROM core_entities_scope_dimension
+            WHERE dimension_slug = 'environment' AND value_slug = 'production'
+          ),
+          ordered AS (
+            SELECT d.id AS deployment_id, d.scope_id, d.release_id, d.created_at,
+                   s.application_id,
+                   groupArray(d.release_id) OVER (PARTITION BY d.scope_id ORDER BY d.created_at ROWS BETWEEN 10 PRECEDING AND 1 PRECEDING) AS prev10,
+                   lagInFrame(d.release_id) OVER (PARTITION BY d.scope_id ORDER BY d.created_at) AS from_release_id
+            FROM core_entities_deployment d FINAL
+            JOIN core_entities_scope s FINAL ON s.id = d.scope_id
+            WHERE d.status = 'finalized' AND d.scope_id IN (SELECT scope_id FROM prod_scopes)
+          )
+          SELECT deployment_id, application_id, scope_id, release_id, from_release_id,
+                 has(prev10, release_id) AS previously_deployed
+          FROM ordered
+          WHERE created_at > now() - INTERVAL ${days} DAY
+            AND (${appId} = 0 OR application_id = ${appId})
+          ORDER BY created_at`;
+
+        const lakeRes = await fetch("https://api.nullplatform.com/data/lake/query", {
+          method: "POST",
+          headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" },
+          body: JSON.stringify({ query: sql }),
+        });
+        stage = "lake";
+        const lakeText = await lakeRes.text();
+        if (!lakeRes.ok) return { stage, error: "http " + lakeRes.status, sample: lakeText.slice(0, 200) };
+        // Lake responds TSV without headers, columns in SELECT order.
+        const COLS = ["deployment_id", "application_id", "scope_id", "release_id", "from_release_id", "previously_deployed"];
+        const rows = lakeText.trim().split("\n").filter(Boolean).map((line) => {
+          const parts = line.split("\t");
+          const r = {};
+          for (let i = 0; i < COLS.length; i++) r[COLS[i]] = parts[i];
+          return r;
+        });
+
+        // Idempotent + convergent: skip deploys whose metadata.change already
+        // exists, so re-running the backfill only fires what's missing. A
+        // large fresh run saturates the sandbox pool (E2B) and some children
+        // die BEFORE the LLM step (zero cost) — just re-fire this webhook
+        // until it reports fired=0.
+        let fired = 0, firstDeploys = 0, errors = 0, skippedDone = 0;
+        for (const r of rows) {
+          const metaRes = await fetch(`https://api.nullplatform.com/metadata/deployment/${r.deployment_id}/change`, {
+            headers: { authorization: `Bearer ${npToken}` },
+          });
+          if (metaRes.ok) { skippedDone++; continue; }
+          const from = String(r.from_release_id || "");
+          // "\N" = ClickHouse NULL in TSV (scope's first deploy ever).
+          // Fire with the explicit "none" sentinel: the analysis runs the
+          // first-deploy path (no diff, first_deploy floor) instead of
+          // wrongly diffing against TODAY's current release.
+          const isFirst = !from || from === "0" || from === "\\N";
+          if (isFirst) firstDeploys++;
+          const body = {
+            kind: "deploy-change-analysis",
+            runId: "backfill-" + r.deployment_id,
+            itemId: "change_analysis",
+            callbackUrl: "https://echo.free.beeceptor.com/checklist-callback",
+            callbackToken: "backfill",
+            approvalRequestId: 0,
+            inputs: {
+              application_id: r.application_id,
+              release_id: r.release_id,
+              scope_id: r.scope_id,
+              deployment_id: r.deployment_id,
+              from_release_id: isFirst ? "none" : from,
+              previously_deployed: String(r.previously_deployed) === "1",
+            },
+          };
+          const res = await fetch(CHILD_WEBHOOK, {
+            method: "POST",
+            headers: { "content-type": "application/json" },
+            body: JSON.stringify(body),
+          });
+          if (res.ok) fired++; else errors++;
+        }
+        return { total_rows: rows.length, fired, already_done: skippedDone, first_deploys: firstDeploys, webhook_errors: errors };
+
+connections:
+  - { id: c1, from: trigger, to: run }
+  - { id: c2, from: trigger_nightly, to: run }