diff --git a/deploy-governance/README.md b/deploy-governance/README.md new file mode 100644 index 0000000..ba348a0 --- /dev/null +++ b/deploy-governance/README.md @@ -0,0 +1,177 @@ +# deploy-governance + +Deploy governance without a human bottleneck: every production deployment +gets an **automatic change analysis** (what changed, who touched it, how risky +it is), the result gates the deploy through a **risk × criticality matrix**, +and the accumulated data rolls up into **weekly functional summaries** per +application and namespace — release notes nobody had to write. + +This is the suite nullplatform runs on its own organization (dogfood). It +replaced "someone self-approves in Slack" with evidence-based gates. + +## The system + +``` +deploy to production + │ + ▼ (checklist external item, kind: deploy-change-analysis) +┌─────────────────────────┐ per-deploy analysis: +│ deploy-change-analysis │──► diff deployed→candidate release (GitHub compare) +│ (np-checklist-trigger) │ PRs as rich objects (author, collaborators+roles, +└───────────┬─────────────┘ description, LLM one-line summary) + │ deterministic risk signals (db migrations, auth, + │ infra, deps paths) + LLM score bounded by floors + │ light matrix: risk × app criticality → auto/peer/group + ▼ + deployment.metadata.change (the durable artifact everything else reads) + ▲ +┌───────────┴─────────────┐ +│ deploy-backfill │ nightly cron (+ manual webhook): finds prod deploys +│ │ WITHOUT metadata in the last N days and fires the +└─────────────────────────┘ analysis for each — self-healing safety net + │ + ▼ (weekly cron, well after the nightly) +┌─────────────────────────┐ ONE LLM call aggregates the week's already-written +│ deploy-weekly-summary │ per-deploy summaries into app summaries + namespace +└───────────┬─────────────┘ roll-ups. No GitHub calls, no re-analysis. + ▼ + application/deploy_summaries + namespace/deploy_summaries + (rolling 53 weeks; the catalog card shows latest_summary only) +``` + +## Workflows + +| File | Trigger | Purpose | +|---|---|---| +| `deploy-change-analysis.yaml` | `np-checklist-trigger`, kind `deploy-change-analysis` | Resolves the checklist external item on real deploys: analysis → metadata → item resolution with markdown + per-gate results | +| `deploy-change-analysis-manual.yaml` | webhook | Same pipeline behind a plain webhook: smoke tests and backfill children. Payload mirrors the trigger outputs (`runId`, `itemId`, `callbackUrl`, `callbackToken`, `inputs.*`) plus backfill overrides (`from_release_id`, `previously_deployed`) | +| `deploy-backfill.yaml` | cron (nightly) + webhook (manual) | Lake query for prod deploys missing `metadata.change` → fires the manual webhook per deploy. Skip-existing makes it idempotent and convergent | +| `deploy-weekly-summary.yaml` | cron (weekly) | Aggregates the week per app/namespace, one LLM call, writes rolling summaries | + +`templates/checklist-template-cross-validation.yaml` is the target checklist +template: the analysis item (informational) plus a `cross_validation` group +(`aggregation: any`) where already-deployed fastpath, the risk matrix, a +four-eyes pair review, and an admin escalation each unblock the deploy. + +## Setup + +1. **Catalog specs** (one-time): + ```bash + NP_API_KEY=... NP_ORGANIZATION_ID= ./setup/01-catalog-specs.sh + ``` + Then classify applications (`governance.criticality`, editable in the UI) + and map users (`identity.github_username`). + +2. **Config entries** (never in YAML): + + | Entry | Kind | Scope | Used by | + |---|---|---|---| + | `NP_API_KEY` | secret | each workflow | NP API + lake access | + | `GITHUB_TOKEN` | secret | analysis workflows | compare/PR/review reads | + | `NP_ORGANIZATION_ID` | var | analysis workflows | trigger NRN + user listing | + | `CHANGE_ANALYSIS_WEBHOOK` | secret | backfill | activated webhook URL of the manual variant (token-bearing → secret) | + + Note: a workflow's `path:` does **not** inherit folder config — scope + entries to each workflow. + +3. **Publish** (`npx np-workflow publish --alias live`), activate, then + set `CHANGE_ANALYSIS_WEBHOOK` to the URL minted for the manual variant + (`GET /workflows/triggers?workflowId=...`). + +4. **Backfill history**: `POST` the backfill webhook with + `{"days": 14, "application_id": 0}` (0 = all apps). Re-fire until it + reports `fired: 0` — see gotchas. + +## Metadata contract: `deployment.metadata.change` + +The durable artifact of the suite. Its specification carries a **full JSON +Schema** (`specs/deployment-change.spec.json`, upserted by the setup script) +so the dashboard renders it schema-driven instead of dumping a JSON blob. + +What the analysis writes per deploy (all levels `additionalProperties: true`): + +| Field | Shape | Notes | +|---|---|---| +| `risk` | `low \| medium \| high` | LLM score bounded by deterministic floors | +| `short_summary` / `summary_md` | string / markdown | one-liner + full narrative (PRs, participants, rationale) | +| `risk_rationale`, `risk_floors_applied` | string, string[] | floors: `db_migration`, `auth_change`, `first_deploy` | +| `change_categories` | `[{category, count, notes}]` | LLM taxonomy (feature/bugfix/security/…); `category` is deliberately a free string in the schema so historical docs never fail write-validation | +| `breaking`, `hotfix` | boolean | only present when true | +| `approval` | `{mode, criticality}` | matrix decision: `auto \| fastpath \| par \| grupo` | +| `from_release` / `to_release` | `{id, semver, commit_sha}` | `from_release` is `null` on first deploys | +| `releases_between`, `previously_deployed` | int, bool | accumulation + rollback fastpath | +| `signals` | sizes + `sensitive_paths{}` | deterministic inputs to the risk score | +| `prs` | rich PR objects | number/title/author/summary/size + per-PR `ai {used, level}` | +| `participants` | `[{github, np_user_id?, roles}]` | `np_user_id` null until mapped via `user/identity` | +| `ai_usage` | `{prs_ai, prs_total, commits_ai, commits_total}` | declared-AI lower bound | + +**Visibility contract**: the schema declares `visibleOn: ["read"]` at the +root — the document renders on the **deployment detail** only. It never +becomes deployment-list columns (nothing is marked `visibleOn: list`, which +list columns require per property) and never appears in create/update forms +(it is machine-written). Raw payloads (`commits`, `files`, long PR +descriptions, plumbing ids) are intentionally **not declared** as properties: +`additionalProperties: true` keeps accepting them on writes, but the +schema-driven UI does not render them — `summary_md` already narrates that +content. + +**Evolving the schema**: the metadata service validates every write against +it, so a stricter schema can brick the analysis pipeline. Before changing +`specs/deployment-change.spec.json`, validate a sample of real stored docs +against the new schema (AJV 8, `strict: false` — same as the service), then +re-run `setup/01-catalog-specs.sh` (it PATCHes the existing spec in place). + +## Cron layout + +- Backfill: nightly (e.g. `30 1 * * *`), window 3 days — self-heals gaps. +- Weekly summary: e.g. Mondays `0 9 * * 1` — hours AFTER the nightly, so the + closing week is fully analyzed before it is summarized. + +## Production gotchas this suite encodes + +- **Trigger config is literal**: `${{ vars.* }}` does NOT resolve inside a + trigger's `config` at activation (steps resolve at run time; triggers do + not) — the `np-checklist-trigger` `nrn` must be a literal, or channel + creation fails with an opaque 401. Activation also needs a caller with + `notification_channel` permissions (the activate call's bearer is the + actor for channel management). + +- **Sandbox pool saturation**: firing 100+ analysis children at once exhausts + the code-exec sandbox pool; children die *before* the LLM step (zero cost). + The dispatcher skips deploys that already have metadata, so re-firing + converges instead of re-paying. Failures show as `SANDBOX_NOT_AVAILABLE` + or as executions that "completed" through their failure-resolve fallback — + check step statuses, not execution status. +- **GitHub reads are parallel**: per-commit PR lookups and per-PR reviews run + in chunks of 10. Sequential, a 100-commit diff exceeds the sandbox budget. +- **First deploys**: the lake's TSV `NULL` (`\N`) for `lag()` means "first + deploy of this scope". The dispatcher sends the explicit `"none"` sentinel; + the analysis then keeps `from = null` instead of wrongly diffing against + *today's* current release. +- **Agent prompt size**: the weekly summary sends a compact `llm_view` + (summaries + PR titles only) — the full gathered object on a busy week + (~180KB) crashes the agent runner. Keep agent prompts under ~100KB. +- **Agent tool detours**: data-in → structured-out agent steps need + "do NOT use tools, respond directly" in the system prompt and enough + `maxIterations` headroom (15), or the model burns its turns exploring. +- **Metadata writes are schema-validated** (with AJV type coercion, so + mismatches produce confusing errors like `prs/0 must be object`). Each + spec must declare exactly what its workflow writes. +- **Catalog UI keys** live *inside* schema properties: `visibleOn` + (`create|read|update|list`; `[]` hides a field from the UI while keeping + it via API), `uiSchema`, `tag`. The dashboard card renders the `read` + context — this suite shows `latest_summary` only and keeps `weeks` as + API/lake data. +- **Lake over HTTP**: `POST /data/lake/query {query}` returns *headerless* + TSV — parse positionally, expect numbers as strings, and add `FINAL` to + versioned tables or rows duplicate. + +## Tests + +The suites' usual plugin-level stubbing does not apply here: this suite's +I/O happens inside `code-exec` sandboxes (raw `fetch` to the NP API, the +lake and GitHub), not through stubbable integration plugins. Validate with +`npx np-workflow validate ` (all four pass, including the dual graph +pass) — behavioral coverage comes from the manual-webhook variant, which +runs the full pipeline against a real deployment without touching any +checklist run. diff --git a/deploy-governance/deploy-backfill.yaml b/deploy-governance/deploy-backfill.yaml new file mode 100644 index 0000000..c6ac517 --- /dev/null +++ b/deploy-governance/deploy-backfill.yaml @@ -0,0 +1,168 @@ +# Deploy Backfill — computes deployment.metadata.change for historical +# production deploys by firing the deploy-change-analysis-manual webhook +# once per deploy. Single code-exec step: lake query + payload build + +# webhook fan-out (children run as independent engine executions). +# +# Fire manually: +# POST /checklist/deploy/backfill +# body: {"days": 15, "application_id": } # application_id: 0 = all apps +# +# NOTE: this engine does not resolve `${{ steps.X.outputs.* }}` inside step +# `inputs:` (arrives empty in the sandbox) — hence the single-step design. +id: deploy-backfill +key: deploy-backfill +name: "Deploy Backfill (change analysis)" +description: > + Queries the lake for finalized production deployments in the last N days, + computes each deploy's previous release at that point in time, and fires + the change-analysis workflow webhook per deploy with from_release_id / + previously_deployed overrides. +semantic_version: 1.2.0 +path: "/checklist/deploy" + +inputs: + triggerPayload: + type: object + default: {} + +steps: + - id: trigger + type: trigger + pluginType: webhook + name: "Manual backfill trigger" + config: + path: /checklist/deploy/backfill + method: POST + mode: start + + # Nightly safety net until the checklist gates every prod deploy: catches + # any production deployment that finished without change metadata (deploys + # while the pilot action is detached, workflow hiccups, etc.). The 3-day + # window + skip-existing makes it self-healing and idempotent. + - id: trigger_nightly + type: trigger + pluginType: cron + name: "Nightly backfill (01:30 AR, last 3 days)" + config: + schedule: "30 1 * * *" + timezone: "America/Argentina/Buenos_Aires" + + - id: run + type: module + pluginType: code-exec + name: "Lake pairs + fan-out webhooks" + inputs: + days: "${{ workflow.inputs.body.days }}" + application_id: "${{ workflow.inputs.body.application_id }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + child_webhook: "${{ secrets.CHANGE_ANALYSIS_WEBHOOK }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + // Default 3: the nightly cron fires with no body — a 3-day window + // self-heals weekend gaps. Manual webhook calls pass days explicitly. + const days = Number($item.days) || 3; + const appId = Number($item.application_id) || 0; + const npApiKey = $item.np_api_key; + // Activated webhook URL of deploy-change-analysis-manual (token-bearing, + // minted at alias activation) — configured as a workflow var, never in YAML. + const CHILD_WEBHOOK = $item.child_webhook; + if (!CHILD_WEBHOOK) return { stage: "config", error: "CHANGE_ANALYSIS_WEBHOOK var is not set" }; + + let stage = "token"; + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: npApiKey }), + }); + const tokText = await tokRes.text(); + let npToken; + try { npToken = JSON.parse(tokText).access_token; } + catch (e) { return { stage, error: String(e), sample: tokText.slice(0, 200) }; } + if (!npToken) return { stage, error: "no access_token", sample: tokText.slice(0, 200) }; + + const sql = ` + WITH prod_scopes AS ( + SELECT DISTINCT scope_id FROM core_entities_scope_dimension + WHERE dimension_slug = 'environment' AND value_slug = 'production' + ), + ordered AS ( + SELECT d.id AS deployment_id, d.scope_id, d.release_id, d.created_at, + s.application_id, + groupArray(d.release_id) OVER (PARTITION BY d.scope_id ORDER BY d.created_at ROWS BETWEEN 10 PRECEDING AND 1 PRECEDING) AS prev10, + lagInFrame(d.release_id) OVER (PARTITION BY d.scope_id ORDER BY d.created_at) AS from_release_id + FROM core_entities_deployment d FINAL + JOIN core_entities_scope s FINAL ON s.id = d.scope_id + WHERE d.status = 'finalized' AND d.scope_id IN (SELECT scope_id FROM prod_scopes) + ) + SELECT deployment_id, application_id, scope_id, release_id, from_release_id, + has(prev10, release_id) AS previously_deployed + FROM ordered + WHERE created_at > now() - INTERVAL ${days} DAY + AND (${appId} = 0 OR application_id = ${appId}) + ORDER BY created_at`; + + const lakeRes = await fetch("https://api.nullplatform.com/data/lake/query", { + method: "POST", + headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" }, + body: JSON.stringify({ query: sql }), + }); + stage = "lake"; + const lakeText = await lakeRes.text(); + if (!lakeRes.ok) return { stage, error: "http " + lakeRes.status, sample: lakeText.slice(0, 200) }; + // Lake responds TSV without headers, columns in SELECT order. + const COLS = ["deployment_id", "application_id", "scope_id", "release_id", "from_release_id", "previously_deployed"]; + const rows = lakeText.trim().split("\n").filter(Boolean).map((line) => { + const parts = line.split("\t"); + const r = {}; + for (let i = 0; i < COLS.length; i++) r[COLS[i]] = parts[i]; + return r; + }); + + // Idempotent + convergent: skip deploys whose metadata.change already + // exists, so re-running the backfill only fires what's missing. A + // large fresh run saturates the sandbox pool (E2B) and some children + // die BEFORE the LLM step (zero cost) — just re-fire this webhook + // until it reports fired=0. + let fired = 0, firstDeploys = 0, errors = 0, skippedDone = 0; + for (const r of rows) { + const metaRes = await fetch(`https://api.nullplatform.com/metadata/deployment/${r.deployment_id}/change`, { + headers: { authorization: `Bearer ${npToken}` }, + }); + if (metaRes.ok) { skippedDone++; continue; } + const from = String(r.from_release_id || ""); + // "\N" = ClickHouse NULL in TSV (scope's first deploy ever). + // Fire with the explicit "none" sentinel: the analysis runs the + // first-deploy path (no diff, first_deploy floor) instead of + // wrongly diffing against TODAY's current release. + const isFirst = !from || from === "0" || from === "\\N"; + if (isFirst) firstDeploys++; + const body = { + kind: "deploy-change-analysis", + runId: "backfill-" + r.deployment_id, + itemId: "change_analysis", + callbackUrl: "https://echo.free.beeceptor.com/checklist-callback", + callbackToken: "backfill", + approvalRequestId: 0, + inputs: { + application_id: r.application_id, + release_id: r.release_id, + scope_id: r.scope_id, + deployment_id: r.deployment_id, + from_release_id: isFirst ? "none" : from, + previously_deployed: String(r.previously_deployed) === "1", + }, + }; + const res = await fetch(CHILD_WEBHOOK, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }); + if (res.ok) fired++; else errors++; + } + return { total_rows: rows.length, fired, already_done: skippedDone, first_deploys: firstDeploys, webhook_errors: errors }; + +connections: + - { id: c1, from: trigger, to: run } + - { id: c2, from: trigger_nightly, to: run } diff --git a/deploy-governance/deploy-change-analysis-manual.yaml b/deploy-governance/deploy-change-analysis-manual.yaml new file mode 100644 index 0000000..cccc7bc --- /dev/null +++ b/deploy-governance/deploy-change-analysis-manual.yaml @@ -0,0 +1,672 @@ +# Deploy Change Analysis — checklist external item resolver. +# +# Pairs with checklist external items of `kind: deploy-change-analysis`. +# Computes the diff between the currently-deployed release and the candidate +# (origin→destination, accumulating intermediate releases), extracts +# participants (GitHub → NP users via user metadata identity.github_username), +# deterministic risk signals, and an LLM risk score bounded by hard floors. +# Persists everything to deployment metadata `change` and resolves the item +# with a markdown summary. +# +# The item is informational — +# it never blocks a deploy. Gates (risk_gate/group_override) come later. +id: deploy-change-analysis-manual +key: deploy-change-analysis-manual +name: "Deploy Change Analysis (manual test)" +description: > + Analyzes the accumulated change of a deployment (deployed release vs + candidate), scores its risk (signals + LLM with hard floors), writes + deployment.metadata.change and resolves the checklist item with a + markdown summary. Informational: resolves passed unless the analysis + itself crashes, in which case it resolves failed with the error. +semantic_version: 1.0.0 +path: "/checklist/deploy" + +inputs: + triggerPayload: + type: object + default: {} + +steps: + - id: trigger + type: trigger + pluginType: webhook + name: "Manual webhook (same payload shape as np-checklist-trigger outputs)" + config: + path: /checklist/deploy/change-analysis-manual + method: POST + mode: start + + - id: progress + type: module + pluginType: np-checklist-item-progress + name: "Heartbeat: analyzing change" + inputs: + callbackUrl: "${{ workflow.inputs.body.callbackUrl }}" + callbackToken: "${{ workflow.inputs.body.callbackToken }}" + message: "Computing diff between deployed and candidate release…" + + - id: gather + type: module + pluginType: code-exec + name: "Gather change data (NP + GitHub)" + inputs: + application_id: "${{ workflow.inputs.body.inputs.application_id }}" + release_id: "${{ workflow.inputs.body.inputs.release_id }}" + scope_id: "${{ workflow.inputs.body.inputs.scope_id }}" + deployment_id: "${{ workflow.inputs.body.inputs.deployment_id }}" + from_release_id: "${{ workflow.inputs.body.inputs.from_release_id }}" + previously_deployed_override: "${{ workflow.inputs.body.inputs.previously_deployed }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + github_token: "${{ secrets.GITHUB_TOKEN }}" + config: + network: + allowedHosts: ["api.nullplatform.com", "api.github.com"] + code: | + const { application_id, release_id, scope_id, np_api_key, github_token } = $item; + + // --- NP client (api-key → token) --- + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const np = async (path) => { + const r = await fetch(`https://api.nullplatform.com${path}`, { + headers: { authorization: `Bearer ${npToken}` }, + }); + if (!r.ok) throw new Error(`NP GET ${path} → ${r.status}`); + return r.json(); + }; + const gh = async (path) => { + const r = await fetch(`https://api.github.com${path}`, { + headers: { authorization: `Bearer ${github_token}`, accept: "application/vnd.github+json" }, + }); + if (!r.ok) throw new Error(`GitHub ${path} → ${r.status}`); + return r.json(); + }; + + const releaseInfo = async (id) => { + const rel = await np(`/release/${id}`); + const build = await np(`/build/${rel.build_id}`); + return { id: rel.id, semver: rel.semver, created_at: rel.created_at, + commit_sha: build.commit && build.commit.id, branch: build.branch }; + }; + + // Candidate (to) release + currently deployed (from) release + fastpath + const to = await releaseInfo(release_id); + const deps = await np(`/deployment?scope_id=${scope_id}&limit=30`); + const finalized = (deps.results || []) + .filter(d => ["finalized", "active"].includes(d.status)) + .sort((a, b) => new Date(b.created_at) - new Date(a.created_at)); + let previously_deployed = finalized.slice(0, 10) + .some(d => String(d.release_id) === String(release_id)); + const pdOverride = $item.previously_deployed_override; + if (pdOverride !== undefined && pdOverride !== null && pdOverride !== "") { + previously_deployed = String(pdOverride) === "true" || pdOverride === true; + } + // from_release_id: "none" (or ClickHouse TSV null "\N") = explicit + // first deploy — keep from=null, do NOT fall back to the current list + // (for backfills the current list reflects TODAY, not deploy time). + const fromArg = String($item.from_release_id || ""); + let from = null; + if (fromArg && fromArg !== "none" && fromArg !== "\\N" && fromArg !== "0") { + from = await releaseInfo(fromArg); + } else if (!fromArg) { + const current = finalized.find(d => String(d.release_id) !== String(release_id)); + if (current) from = await releaseInfo(current.release_id); + } + + // Repo + const app = await np(`/application/${application_id}`); + const repo = (app.repository_url || "").replace(/^https:\/\/github\.com\//, "").replace(/\.git$/, ""); + if (!repo) throw new Error(`application ${application_id} has no repository_url`); + + // Intermediate releases + let releases_between = 0; + if (from) { + const rels = await np(`/release?application_id=${application_id}&limit=100`); + releases_between = (rels.results || []).filter(r => + new Date(r.created_at) > new Date(from.created_at) && + new Date(r.created_at) < new Date(to.created_at)).length; + } + + // GitHub compare + participants (roles: author / reviewer / approver) + let commits = [], files = [], prs = [], participants = []; + if (!previously_deployed && from && from.commit_sha && to.commit_sha && from.commit_sha !== to.commit_sha) { + const cmp = await gh(`/repos/${repo}/compare/${from.commit_sha}...${to.commit_sha}`); + commits = (cmp.commits || []).map(c => ({ sha: c.sha, message: c.commit.message.split("\n")[0] })); + files = (cmp.files || []).map(f => ({ filename: f.filename, additions: f.additions, deletions: f.deletions })); + // Parallel in chunks: huge diffs (100+ commits) exceeded the 60s + // sandbox budget when these calls ran sequentially. + const inChunks = async (items, size, fn) => { + const out = []; + for (let i = 0; i < items.length; i += size) { + out.push(...await Promise.all(items.slice(i, i + size).map(fn))); + } + return out; + }; + const prMap = new Map(); + const prLists = await inChunks((cmp.commits || []).slice(0, 250), 10, + c => gh(`/repos/${repo}/commits/${c.sha}/pulls`).catch(() => [])); + for (const list of prLists) for (const pr of list) prMap.set(pr.number, pr); + const people = new Map(); + const add = (login, role) => { + if (!login || login.endsWith("[bot]")) return; + if (!people.has(login)) people.set(login, new Set()); + people.get(login).add(role); + }; + for (const c of cmp.commits || []) add(c.author && c.author.login, "author"); + const prArr = [...prMap.values()]; + const reviewLists = await inChunks(prArr, 10, + pr => gh(`/repos/${repo}/pulls/${pr.number}/reviews`).catch(() => [])); + // Lead-time + size inputs: the PR detail carries additions/deletions + // (absent from the simple PR object) and /commits page 1 item 1 is + // the OLDEST commit — the true "first line" even under squash-merge. + const prDetails = await inChunks(prArr, 10, + pr => gh(`/repos/${repo}/pulls/${pr.number}`).catch(() => null)); + const prFirstCommits = await inChunks(prArr, 10, + pr => gh(`/repos/${repo}/pulls/${pr.number}/commits?per_page=1`).catch(() => [])); + prArr.forEach((pr, i) => { + add(pr.user && pr.user.login, "author"); + const prPeople = new Map(); + const addPr = (login, role) => { + if (!login || login.endsWith("[bot]")) return; + if (!prPeople.has(login)) prPeople.set(login, new Set()); + prPeople.get(login).add(role); + }; + addPr(pr.user && pr.user.login, "author"); + for (const rv of reviewLists[i]) { + add(rv.user && rv.user.login, rv.state === "APPROVED" ? "approver" : "reviewer"); + addPr(rv.user && rv.user.login, rv.state === "APPROVED" ? "approver" : "reviewer"); + } + pr._collab = [...prPeople].map(([github, roles]) => ({ github, roles: [...roles] })); + }); + // --- AI usage per PR: commit trailers (Co-Authored-By: Claude/...) + // and PR-body markers. "level" buckets the share of AI commits. + const AI_COMMIT = /co-authored-by:.*?(claude|copilot|cursor|devin|aider|codegen|windsurf)|generated with \[?claude|claude-session:/i; + const AI_BODY = /claude code|claude\.ai\/code|github copilot|devin\.ai|generated with/i; + const prAi = new Map(); + const cList = (cmp.commits || []).slice(0, 250); + cList.forEach((c, i) => { + const isAi = AI_COMMIT.test((c.commit && c.commit.message) || ""); + for (const pr of prLists[i] || []) { + if (!prAi.has(pr.number)) prAi.set(pr.number, { total: 0, ai: 0 }); + const s = prAi.get(pr.number); s.total++; if (isAi) s.ai++; + } + }); + prs = prArr.map((p, pi) => { + const st = prAi.get(p.number) || { total: 0, ai: 0 }; + const det = prDetails[pi]; + const fc = (prFirstCommits[pi] || [])[0]; + const bodyMarker = AI_BODY.test(p.body || ""); + const ratio = st.total ? st.ai / st.total : 0; + const used = st.ai > 0 || bodyMarker; + return { + number: p.number, title: p.title, branch: p.head && p.head.ref, + author: p.user && p.user.login, + collaborators: p._collab || [], + description: (p.body || "").slice(0, 500), + first_commit_at: (fc && fc.commit && fc.commit.author && fc.commit.author.date) || null, + opened_at: p.created_at || null, + merged_at: p.merged_at || null, + size: det ? { additions: det.additions, deletions: det.deletions, files: det.changed_files } : null, + ai: { + used, + level: !used ? "none" : ratio >= 1 ? "full" : ratio >= 0.5 ? "mostly" : "assisted", + commits_ai: st.ai, commits_total: st.total, body_marker: bodyMarker, + }, + }; + }); + participants = [...people].map(([github, roles]) => ({ github, roles: [...roles] })); + } + + // Participants stay as {github, roles}: mapping to platform users + // happens at REPORT time (lake join vs user/identity metadata) — it + // stays current when mappings change and costs zero API calls here. + + // Deterministic signals + const PATTERNS = { + db_migrations: /(^|\/)(migrations?|db\/migrate|liquibase|flyway)\//i, + auth: /(^|\/)(auth|authn|authz|permissions?|iam)\//i, + infra: /(^|\/)(terraform|infra(structure)?|helm|k8s|\.github\/workflows)\/|(^|\/)Dockerfile$/i, + dependencies: /(^|\/)(package(-lock)?\.json|pnpm-lock\.yaml|yarn\.lock|go\.(mod|sum)|requirements\.txt|Gemfile(\.lock)?|pom\.xml)$/i, + }; + const sensitive_paths = {}; + for (const k of Object.keys(PATTERNS)) sensitive_paths[k] = files.some(f => PATTERNS[k].test(f.filename)); + const signals = { + lines_added: files.reduce((n, f) => n + (f.additions || 0), 0), + lines_deleted: files.reduce((n, f) => n + (f.deletions || 0), 0), + files_changed: files.length, + commits: commits.length, + prs: prs.map(p => p.number), + branches: new Set(prs.map(p => p.branch).filter(Boolean)).size, + intermediate_releases: releases_between, + sensitive_paths, + }; + + let criticality = "critical"; + try { + const gmeta = await np(`/metadata/application/${application_id}`); + if (gmeta.governance && gmeta.governance.criticality) criticality = String(gmeta.governance.criticality); + } catch (e) { /* unclassified -> critical */ } + + let deployment_id = $item.deployment_id; + if (!deployment_id) { + const cand = (deps.results || []).find(d => String(d.release_id) === String(release_id)); + deployment_id = cand && cand.id; + } + + return { + deployment_id, + criticality, + from_release: from ? { id: from.id, semver: from.semver, commit_sha: from.commit_sha } : null, + to_release: { id: to.id, semver: to.semver, commit_sha: to.commit_sha }, + previously_deployed, + releases_between, + commits, files, prs, participants, signals, + ai_usage: { + prs_ai: prs.filter(p => p.ai && p.ai.used).length, + prs_total: prs.length, + commits_ai: prs.reduce((n, p) => n + ((p.ai && p.ai.commits_ai) || 0), 0), + commits_total: prs.reduce((n, p) => n + ((p.ai && p.ai.commits_total) || 0), 0), + }, + }; + errorHandling: + fallbackStep: resolve_failure + + - id: progress_gathered + type: module + pluginType: np-checklist-item-progress + name: "Heartbeat: diff computed" + inputs: + callbackUrl: "${{ workflow.inputs.body.callbackUrl }}" + callbackToken: "${{ workflow.inputs.body.callbackToken }}" + message: "Diff computed: ${{ steps.gather.outputs.signals.commits }} commits, ${{ steps.gather.outputs.signals.files_changed }} files, ${{ steps.gather.outputs.releases_between }} intermediate releases. Scoring risk…" + + - id: risk_agent + type: module + pluginType: claude-code-agent + name: "Score change risk" + config: + model: claude-opus-4-8 + maxIterations: 3 + systemPrompt: | + You assess deployment risk for nullplatform's own production deploys. + Rules of thumb: config/copy tweaks and small isolated fixes are low; + new features, schema or dependency changes are medium; anything + touching auth, data migrations, infra, or very large / multi-branch + accumulations is high. Be concise and concrete: name the riskiest + part of the change in the rationale. + + The summary must be FUNCTIONAL: describe what this deploy changes in + behavior or capability (features shipped, bugs fixed, what users or + operators will notice), NOT which files were touched — file and line + detail is already shown as structured data next to your summary. + Name a code component only when it is essential to understand the risk. + userPrompt: | + Assess this deployment change. + + Signals: ${{ steps.gather.outputs.signals }} + Commit messages: ${{ steps.gather.outputs.commits }} + PR titles: ${{ steps.gather.outputs.prs }} + Files: ${{ steps.gather.outputs.files }} + + Produce: + - risk: low | medium | high + - rationale: ≤50 words, name the riskiest concrete element + - summary_md: ≤120-word markdown summary of what this deploy changes + FUNCTIONALLY (features, fixes, behavior), for the human who approves + it. Do not list files or line counts. + - short_summary: ONE plain sentence (max 15 words, no markdown) naming + the essential functional change — used where UI space is tight. + - pr_summaries: for EACH PR in the input, a one-line functional summary + (≤25 words) of what that PR ships. + outputSchema: + type: object + required: [risk, rationale, summary_md, short_summary] + properties: + risk: { type: string, enum: [low, medium, high] } + rationale: { type: string } + summary_md: { type: string } + short_summary: { type: string } + pr_summaries: + type: array + items: + type: object + required: [number, summary] + properties: + number: { type: integer } + summary: { type: string } + errorHandling: + fallbackStep: assemble_heuristic + + # LLM path: floors + assemble the final change object. + - id: assemble + type: module + pluginType: code-exec + name: "Apply floors + assemble metadata" + inputs: + change: "${{ steps.gather.outputs }}" + llm: "${{ steps.risk_agent.outputs }}" + config: + code: | + const change = $item.change; + const llm = $item.llm || {}; + const ORDER = { low: 0, medium: 1, high: 2 }; + const floors = []; + const sp = (change.signals && change.signals.sensitive_paths) || {}; + if (sp.db_migrations) floors.push("db_migration"); + if (sp.auth) floors.push("auth_change"); + if (!change.from_release) floors.push("first_deploy"); + const heuristic = (() => { + const s = change.signals || {}; + let n = 0; + if ((s.lines_added || 0) + (s.lines_deleted || 0) > 500) n++; + if ((s.files_changed || 0) > 20) n++; + if ((s.intermediate_releases || 0) > 2) n++; + if (sp.dependencies) n++; + if (sp.infra) n++; + if (sp.db_migrations || sp.auth) n += 2; + return n >= 3 ? "high" : n >= 1 ? "medium" : "low"; + })(); + const scored = llm.risk || heuristic; + const floored = floors.length ? "medium" : "low"; + const risk = ORDER[scored] >= ORDER[floored] ? scored : floored; + const summary_md = llm.summary_md || + `${(change.commits || []).length} commits, ${(change.signals || {}).files_changed || 0} files: ` + + ((change.prs || []).map(p => p.title).join("; ") || (change.commits || []).slice(0, 5).map(c => c.message).join("; ") || "no commit data"); + const rationale = llm.rationale || "Heuristic score (LLM unavailable)."; + const short_summary = llm.short_summary || + (summary_md.split(/[.\n]/)[0] || "").replace(/[*_#`]/g, "").trim().slice(0, 120); + const RANK = { mission_critical: 1, critical: 2, important: 3, standard: 4, internal: 5 }; + const critName = change.criticality || "critical"; + const rank = RANK[critName] || 2; + let mode; + if (change.previously_deployed) mode = "fastpath"; + else if (risk === "low") mode = "auto"; + else if (risk === "medium") mode = rank <= 2 ? "par" : "auto"; + else mode = rank === 1 ? "grupo" : rank <= 3 ? "par" : "auto"; + const auto = mode === "auto" || mode === "fastpath"; + const risk_gate_status = auto ? "passed" : "failed"; + const risk_gate_message = auto + ? (mode === "fastpath" ? "Release already deployed to this scope recently - rollback fastpath" : "Auto-approved: risk " + risk + " x criticality " + critName) + : "Matrix requires " + mode + " approval (risk " + risk + " x criticality " + critName + ") - falls back to manual review until Slack approvals land"; + const participantsLine = (change.participants || []) + .map(p => `${p.github} [${p.roles.join(", ")}]`).join(", ") || "none detected"; + const prSummaries = new Map(((llm && llm.pr_summaries) || []).map(p => [p.number, p.summary])); + const prs_enriched = (change.prs || []).map(p => Object.assign({}, p, { + summary: prSummaries.get(p.number) || (p.description ? p.description.split("\n")[0].slice(0, 160) : null), + })); + const prLines = prs_enriched.map(p => { + const others = (p.collaborators || []).filter(c => c.github !== p.author) + .map(c => `${c.github} (${c.roles.join("/")})`).join(", "); + const aiTag = p.ai && p.ai.used ? ` · 🤖 AI ${p.ai.level}` : ""; + return `- **#${p.number} ${p.title}** — ${p.author}${others ? " · con " + others : ""}${aiTag}${p.summary ? `\n ${p.summary}` : ""}`; + }).join("\n"); + const markdown = [ + `### Change analysis — risk: **${risk.toUpperCase()}**${floors.length ? ` _(floors: ${floors.join(", ")})_` : ""}`, + "", + change.previously_deployed ? "♻️ **This release was already deployed to this scope recently (rollback fastpath candidate).**\n" : "", + summary_md, + prLines ? "\n**PRs:**\n" + prLines : "", + "", + `**From → to:** ${change.from_release ? change.from_release.semver : "(first deploy)"} → ${change.to_release.semver} (${change.releases_between} releases in between)`, + `**Size:** ${change.signals.lines_added}+/${change.signals.lines_deleted}- across ${change.signals.files_changed} files, ${change.signals.commits} commits, ${(change.prs || []).length} PRs`, + `**Participants:** ${participantsLine}`, + `**Why:** ${rationale}`, + ].join("\n"); + const gate_results = [ + { item: "already_deployed", + status: change.previously_deployed ? "passed" : "failed", + message: change.previously_deployed + ? "Release deployed to this scope within the last 10 finalized deploys - rollback fastpath" + : "Release not among the last 10 finalized deploys of this scope" }, + { item: "risk_matrix", status: auto ? "passed" : "failed", message: risk_gate_message }, + { item: "risk_gate", status: risk_gate_status, message: risk_gate_message }, + ]; + return { + metadata: Object.assign({}, change, { + prs: prs_enriched, + risk, risk_rationale: rationale, risk_floors_applied: floors, summary_md, short_summary, + approval: { mode, criticality: critName }, + }), + markdown: markdown + "\n**Decision:** " + mode, + risk, gate_results, decision_mode: mode, + }; + + # Heuristic path (LLM failed): dedicated node so joins stay single-predecessor. + - id: assemble_heuristic + type: module + pluginType: code-exec + name: "Assemble metadata (heuristic, LLM failed)" + inputs: + change: "${{ steps.gather.outputs }}" + config: + code: | + const change = $item.change; + const ORDER = { low: 0, medium: 1, high: 2 }; + const floors = []; + const sp = (change.signals && change.signals.sensitive_paths) || {}; + if (sp.db_migrations) floors.push("db_migration"); + if (sp.auth) floors.push("auth_change"); + if (!change.from_release) floors.push("first_deploy"); + const s = change.signals || {}; + let n = 0; + if ((s.lines_added || 0) + (s.lines_deleted || 0) > 500) n++; + if ((s.files_changed || 0) > 20) n++; + if ((s.intermediate_releases || 0) > 2) n++; + if (sp.dependencies) n++; + if (sp.infra) n++; + if (sp.db_migrations || sp.auth) n += 2; + const scored = n >= 3 ? "high" : n >= 1 ? "medium" : "low"; + const floored = floors.length ? "medium" : "low"; + const risk = ORDER[scored] >= ORDER[floored] ? scored : floored; + const RANK = { mission_critical: 1, critical: 2, important: 3, standard: 4, internal: 5 }; + const critName = change.criticality || "critical"; + const rank = RANK[critName] || 2; + let mode; + if (change.previously_deployed) mode = "fastpath"; + else if (risk === "low") mode = "auto"; + else if (risk === "medium") mode = rank <= 2 ? "par" : "auto"; + else mode = rank === 1 ? "grupo" : rank <= 3 ? "par" : "auto"; + const auto = mode === "auto" || mode === "fastpath"; + const risk_gate_status = auto ? "passed" : "failed"; + const risk_gate_message = auto + ? (mode === "fastpath" ? "Release already deployed to this scope recently - rollback fastpath" : "Auto-approved: risk " + risk + " x criticality " + critName) + : "Matrix requires " + mode + " approval (risk " + risk + " x criticality " + critName + ") - falls back to manual review until Slack approvals land"; + const rationale = "Heuristic score (LLM step failed): size, accumulation and sensitive-path signals only."; + const summary_md = `${(change.commits || []).length} commits, ${s.files_changed || 0} files: ` + + ((change.prs || []).map(p => p.title).join("; ") || "no commit data"); + const short_summary = ((change.prs || [])[0] ? (change.prs || [])[0].title : `${(change.commits || []).length} commits`).slice(0, 120); + const markdown = `### Change analysis — risk: **${risk.toUpperCase()}** _(heuristic)_\n\n${summary_md}\n\n**Why:** ${rationale}`; + const gate_results = [ + { item: "already_deployed", + status: change.previously_deployed ? "passed" : "failed", + message: change.previously_deployed + ? "Release deployed to this scope within the last 10 finalized deploys - rollback fastpath" + : "Release not among the last 10 finalized deploys of this scope" }, + { item: "risk_matrix", status: auto ? "passed" : "failed", message: risk_gate_message }, + { item: "risk_gate", status: risk_gate_status, message: risk_gate_message }, + ]; + return { + metadata: Object.assign({}, change, { + risk, risk_rationale: rationale, risk_floors_applied: floors, summary_md, short_summary, + approval: { mode, criticality: critName }, + }), + markdown: markdown + "\n**Decision:** " + mode, + risk, gate_results, decision_mode: mode, + }; + + # ── Success tail (LLM path) ──────────────────────────────────────────── + - id: write_metadata + type: module + pluginType: code-exec + name: "Persist deployment.metadata.change (upsert)" + inputs: + deployment_id: "${{ steps.gather.outputs.deployment_id }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + metadata: "${{ steps.assemble.outputs.metadata }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { deployment_id, np_api_key, metadata } = $item; + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const url = `https://api.nullplatform.com/metadata/deployment/${deployment_id}/change`; + const call = (method) => fetch(url, { + method, + headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" }, + body: JSON.stringify(metadata), + }); + let res = await call("POST"); + if (!res.ok) { + const text = await res.text(); + if (text.includes("already exists")) res = await call("PATCH"); + else throw new Error(`metadata POST → ${res.status}: ${text}`); + } + if (!res.ok) throw new Error(`metadata PATCH → ${res.status}: ${await res.text()}`); + return { written: true }; + + + - id: resolve + type: module + pluginType: code-exec + name: "Resolve items (change_analysis + risk_gate)" + inputs: + callbackUrl: "${{ workflow.inputs.body.callbackUrl }}" + callbackToken: "${{ workflow.inputs.body.callbackToken }}" + out: "${{ steps.assemble.outputs }}" + config: + network: + allowedHosts: ["api.nullplatform.com", "echo.free.beeceptor.com"] + code: | + const { callbackUrl, callbackToken, out } = $item; + const H = { authorization: `Bearer ${callbackToken}`, "content-type": "application/json" }; + const post = (url, body) => fetch(url, { method: "POST", headers: H, body: JSON.stringify(body) }); + const patch = (url, body) => fetch(url, { method: "PATCH", headers: H, body: JSON.stringify(body) }); + await post(`${callbackUrl}/log`, { level: "info", message: `Risk: ${out.risk} - ${out.metadata.risk_rationale}` }).catch(() => null); + await post(`${callbackUrl}/log`, { level: "info", message: `Matrix decision: ${out.decision_mode} (criticality ${out.metadata.criticality || "critical"})` }).catch(() => null); + let r = await patch(callbackUrl, { status: "passed", message: `Risk ${out.risk}: ${out.metadata.short_summary || "change analyzed"}`, details: { markdown: out.markdown } }); + if (!r.ok) throw new Error(`change_analysis resolve -> ${r.status}: ${await r.text()}`); + // Sibling gates (cross_validation group, template v6). Tolerated when + // the item is absent (404), when the token is item-scoped pre-#260 + // (401), or when the run already resolved (409). + const results = {}; + for (const g of out.gate_results || []) { + const url = callbackUrl.replace(/\/items\/[^/]+$/, `/items/${g.item}`); + const rr = await patch(url, { status: g.status, message: g.message }); + results[g.item] = rr.ok ? g.status : `skipped_${rr.status}`; + if (!rr.ok && ![401, 404, 409].includes(rr.status)) throw new Error(`${g.item} resolve -> ${rr.status}: ${await rr.text()}`); + } + return { resolved: true, gates: results }; + + + # ── Heuristic tail (LLM failed; dedicated nodes, no mixed joins) ────── + - id: write_metadata_h + type: module + pluginType: code-exec + name: "Persist deployment.metadata.change (heuristic, upsert)" + inputs: + deployment_id: "${{ steps.gather.outputs.deployment_id }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + metadata: "${{ steps.assemble_heuristic.outputs.metadata }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { deployment_id, np_api_key, metadata } = $item; + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const url = `https://api.nullplatform.com/metadata/deployment/${deployment_id}/change`; + const call = (method) => fetch(url, { + method, + headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" }, + body: JSON.stringify(metadata), + }); + let res = await call("POST"); + if (!res.ok) { + const text = await res.text(); + if (text.includes("already exists")) res = await call("PATCH"); + else throw new Error(`metadata POST → ${res.status}: ${text}`); + } + if (!res.ok) throw new Error(`metadata PATCH → ${res.status}: ${await res.text()}`); + return { written: true }; + + + - id: resolve_h + type: module + pluginType: code-exec + name: "Resolve items (change_analysis + risk_gate)" + inputs: + callbackUrl: "${{ workflow.inputs.body.callbackUrl }}" + callbackToken: "${{ workflow.inputs.body.callbackToken }}" + out: "${{ steps.assemble_heuristic.outputs }}" + config: + network: + allowedHosts: ["api.nullplatform.com", "echo.free.beeceptor.com"] + code: | + const { callbackUrl, callbackToken, out } = $item; + const H = { authorization: `Bearer ${callbackToken}`, "content-type": "application/json" }; + const post = (url, body) => fetch(url, { method: "POST", headers: H, body: JSON.stringify(body) }); + const patch = (url, body) => fetch(url, { method: "PATCH", headers: H, body: JSON.stringify(body) }); + await post(`${callbackUrl}/log`, { level: "info", message: `Risk: ${out.risk} - ${out.metadata.risk_rationale}` }).catch(() => null); + await post(`${callbackUrl}/log`, { level: "info", message: `Matrix decision: ${out.decision_mode} (criticality ${out.metadata.criticality || "critical"})` }).catch(() => null); + let r = await patch(callbackUrl, { status: "passed", message: `Risk ${out.risk}: ${out.metadata.short_summary || "change analyzed"}`, details: { markdown: out.markdown } }); + if (!r.ok) throw new Error(`change_analysis resolve -> ${r.status}: ${await r.text()}`); + // Sibling gates (cross_validation group, template v6). Tolerated when + // the item is absent (404), when the token is item-scoped pre-#260 + // (401), or when the run already resolved (409). + const results = {}; + for (const g of out.gate_results || []) { + const url = callbackUrl.replace(/\/items\/[^/]+$/, `/items/${g.item}`); + const rr = await patch(url, { status: g.status, message: g.message }); + results[g.item] = rr.ok ? g.status : `skipped_${rr.status}`; + if (!rr.ok && ![401, 404, 409].includes(rr.status)) throw new Error(`${g.item} resolve -> ${rr.status}: ${await rr.text()}`); + } + return { resolved: true, gates: results }; + + + # Reached only via gather's `error_handling.fallback_step`: the analysis + # itself crashed (NP/GitHub unavailable, bad repo, etc.). The item is + # informational so this does not block the deploy; the error is surfaced + # verbatim for debugging. + - id: resolve_failure + type: module + pluginType: np-checklist-item-resolve + name: "Resolve as failed (analysis crashed)" + inputs: + callbackUrl: "${{ workflow.inputs.body.callbackUrl }}" + callbackToken: "${{ workflow.inputs.body.callbackToken }}" + status: failed + message: "Change analysis crashed: ${{ steps.gather.error.message }}" + details: + markdown: "The analysis workflow failed before producing a result. Deploy is NOT blocked (informational item). Error: ${{ steps.gather.error.message }}" + +connections: + - { id: c1, from: trigger, to: progress } + - { id: c2, from: progress, to: gather } + - { id: c3, from: gather, to: progress_gathered } + - { id: c3b, from: progress_gathered, to: risk_agent } + - { id: c4, from: risk_agent, to: assemble } + - { id: c5, from: assemble, to: resolve } + - { id: c6, from: resolve, to: write_metadata } + # Dormant edges: keep the fallback tails out of the entry-point set; + # the engine routes to them on terminal failure via fallback_step. + - { id: c7, from: gather, to: resolve_failure, condition: "false" } + - { id: c8, from: risk_agent, to: assemble_heuristic, condition: "false" } + - { id: c9, from: assemble_heuristic, to: resolve_h } + - { id: c10, from: resolve_h, to: write_metadata_h } diff --git a/deploy-governance/deploy-change-analysis.yaml b/deploy-governance/deploy-change-analysis.yaml new file mode 100644 index 0000000..a7100e0 --- /dev/null +++ b/deploy-governance/deploy-change-analysis.yaml @@ -0,0 +1,676 @@ +# Deploy Change Analysis — checklist external item resolver. +# +# Pairs with checklist external items of `kind: deploy-change-analysis`. +# Computes the diff between the currently-deployed release and the candidate +# (origin→destination, accumulating intermediate releases), extracts +# participants (GitHub → NP users via user metadata identity.github_username), +# deterministic risk signals, and an LLM risk score bounded by hard floors. +# Persists everything to deployment metadata `change` and resolves the item +# with a markdown summary. +# +# The item is informational — +# it never blocks a deploy. Gates (risk_gate/group_override) come later. +id: deploy-change-analysis +key: deploy-change-analysis +name: "Deploy Change Analysis" +description: > + Analyzes the accumulated change of a deployment (deployed release vs + candidate), scores its risk (signals + LLM with hard floors), writes + deployment.metadata.change and resolves the checklist item with a + markdown summary. Informational: resolves passed unless the analysis + itself crashes, in which case it resolves failed with the error. +semantic_version: 1.0.0 +path: "/checklist/deploy" + +inputs: + triggerPayload: + type: object + default: {} + +steps: + - id: trigger + type: trigger + pluginType: np-checklist-trigger + name: "On checklist dispatch (deploy-change-analysis)" + config: + pathPrefix: /checklist/deploy/change-analysis + mode: start + kind: deploy-change-analysis + # LITERAL on purpose: trigger config does NOT resolve ${{ vars.* }} at + # activation time — an unresolved nrn makes the channel POST 401. + nrn: "organization=" + npApiKey: "${{ secrets.NP_API_KEY }}" + + - id: progress + type: module + pluginType: np-checklist-item-progress + name: "Heartbeat: analyzing change" + inputs: + callbackUrl: "${{ steps.trigger.outputs.callbackUrl }}" + callbackToken: "${{ steps.trigger.outputs.callbackToken }}" + message: "Computing diff between deployed and candidate release…" + + - id: gather + type: module + pluginType: code-exec + name: "Gather change data (NP + GitHub)" + inputs: + application_id: "${{ steps.trigger.outputs.inputs.application_id }}" + release_id: "${{ steps.trigger.outputs.inputs.release_id }}" + scope_id: "${{ steps.trigger.outputs.inputs.scope_id }}" + deployment_id: "${{ steps.trigger.outputs.inputs.deployment_id }}" + from_release_id: "${{ steps.trigger.outputs.inputs.from_release_id }}" + previously_deployed_override: "${{ steps.trigger.outputs.inputs.previously_deployed }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + github_token: "${{ secrets.GITHUB_TOKEN }}" + config: + network: + allowedHosts: ["api.nullplatform.com", "api.github.com"] + code: | + const { application_id, release_id, scope_id, np_api_key, github_token } = $item; + + // --- NP client (api-key → token) --- + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const np = async (path) => { + const r = await fetch(`https://api.nullplatform.com${path}`, { + headers: { authorization: `Bearer ${npToken}` }, + }); + if (!r.ok) throw new Error(`NP GET ${path} → ${r.status}`); + return r.json(); + }; + const gh = async (path) => { + const r = await fetch(`https://api.github.com${path}`, { + headers: { authorization: `Bearer ${github_token}`, accept: "application/vnd.github+json" }, + }); + if (!r.ok) throw new Error(`GitHub ${path} → ${r.status}`); + return r.json(); + }; + + const releaseInfo = async (id) => { + const rel = await np(`/release/${id}`); + const build = await np(`/build/${rel.build_id}`); + return { id: rel.id, semver: rel.semver, created_at: rel.created_at, + commit_sha: build.commit && build.commit.id, branch: build.branch }; + }; + + // Candidate (to) release + currently deployed (from) release + fastpath + const to = await releaseInfo(release_id); + const deps = await np(`/deployment?scope_id=${scope_id}&limit=30`); + const finalized = (deps.results || []) + .filter(d => ["finalized", "active"].includes(d.status)) + .sort((a, b) => new Date(b.created_at) - new Date(a.created_at)); + let previously_deployed = finalized.slice(0, 10) + .some(d => String(d.release_id) === String(release_id)); + const pdOverride = $item.previously_deployed_override; + if (pdOverride !== undefined && pdOverride !== null && pdOverride !== "") { + previously_deployed = String(pdOverride) === "true" || pdOverride === true; + } + // from_release_id: "none" (or ClickHouse TSV null "\N") = explicit + // first deploy — keep from=null, do NOT fall back to the current list + // (for backfills the current list reflects TODAY, not deploy time). + const fromArg = String($item.from_release_id || ""); + let from = null; + if (fromArg && fromArg !== "none" && fromArg !== "\\N" && fromArg !== "0") { + from = await releaseInfo(fromArg); + } else if (!fromArg) { + const current = finalized.find(d => String(d.release_id) !== String(release_id)); + if (current) from = await releaseInfo(current.release_id); + } + + // Repo + const app = await np(`/application/${application_id}`); + const repo = (app.repository_url || "").replace(/^https:\/\/github\.com\//, "").replace(/\.git$/, ""); + if (!repo) throw new Error(`application ${application_id} has no repository_url`); + + // Intermediate releases + let releases_between = 0; + if (from) { + const rels = await np(`/release?application_id=${application_id}&limit=100`); + releases_between = (rels.results || []).filter(r => + new Date(r.created_at) > new Date(from.created_at) && + new Date(r.created_at) < new Date(to.created_at)).length; + } + + // GitHub compare + participants (roles: author / reviewer / approver) + let commits = [], files = [], prs = [], participants = []; + if (!previously_deployed && from && from.commit_sha && to.commit_sha && from.commit_sha !== to.commit_sha) { + const cmp = await gh(`/repos/${repo}/compare/${from.commit_sha}...${to.commit_sha}`); + commits = (cmp.commits || []).map(c => ({ sha: c.sha, message: c.commit.message.split("\n")[0] })); + files = (cmp.files || []).map(f => ({ filename: f.filename, additions: f.additions, deletions: f.deletions })); + // Parallel in chunks: huge diffs (100+ commits) exceeded the 60s + // sandbox budget when these calls ran sequentially. + const inChunks = async (items, size, fn) => { + const out = []; + for (let i = 0; i < items.length; i += size) { + out.push(...await Promise.all(items.slice(i, i + size).map(fn))); + } + return out; + }; + const prMap = new Map(); + const prLists = await inChunks((cmp.commits || []).slice(0, 250), 10, + c => gh(`/repos/${repo}/commits/${c.sha}/pulls`).catch(() => [])); + for (const list of prLists) for (const pr of list) prMap.set(pr.number, pr); + const people = new Map(); + const add = (login, role) => { + if (!login || login.endsWith("[bot]")) return; + if (!people.has(login)) people.set(login, new Set()); + people.get(login).add(role); + }; + for (const c of cmp.commits || []) add(c.author && c.author.login, "author"); + const prArr = [...prMap.values()]; + const reviewLists = await inChunks(prArr, 10, + pr => gh(`/repos/${repo}/pulls/${pr.number}/reviews`).catch(() => [])); + // Lead-time + size inputs: the PR detail carries additions/deletions + // (absent from the simple PR object) and /commits page 1 item 1 is + // the OLDEST commit — the true "first line" even under squash-merge. + const prDetails = await inChunks(prArr, 10, + pr => gh(`/repos/${repo}/pulls/${pr.number}`).catch(() => null)); + const prFirstCommits = await inChunks(prArr, 10, + pr => gh(`/repos/${repo}/pulls/${pr.number}/commits?per_page=1`).catch(() => [])); + prArr.forEach((pr, i) => { + add(pr.user && pr.user.login, "author"); + const prPeople = new Map(); + const addPr = (login, role) => { + if (!login || login.endsWith("[bot]")) return; + if (!prPeople.has(login)) prPeople.set(login, new Set()); + prPeople.get(login).add(role); + }; + addPr(pr.user && pr.user.login, "author"); + for (const rv of reviewLists[i]) { + add(rv.user && rv.user.login, rv.state === "APPROVED" ? "approver" : "reviewer"); + addPr(rv.user && rv.user.login, rv.state === "APPROVED" ? "approver" : "reviewer"); + } + pr._collab = [...prPeople].map(([github, roles]) => ({ github, roles: [...roles] })); + }); + // --- AI usage per PR: commit trailers (Co-Authored-By: Claude/...) + // and PR-body markers. "level" buckets the share of AI commits. + const AI_COMMIT = /co-authored-by:.*?(claude|copilot|cursor|devin|aider|codegen|windsurf)|generated with \[?claude|claude-session:/i; + const AI_BODY = /claude code|claude\.ai\/code|github copilot|devin\.ai|generated with/i; + const prAi = new Map(); + const cList = (cmp.commits || []).slice(0, 250); + cList.forEach((c, i) => { + const isAi = AI_COMMIT.test((c.commit && c.commit.message) || ""); + for (const pr of prLists[i] || []) { + if (!prAi.has(pr.number)) prAi.set(pr.number, { total: 0, ai: 0 }); + const s = prAi.get(pr.number); s.total++; if (isAi) s.ai++; + } + }); + prs = prArr.map((p, pi) => { + const st = prAi.get(p.number) || { total: 0, ai: 0 }; + const det = prDetails[pi]; + const fc = (prFirstCommits[pi] || [])[0]; + const bodyMarker = AI_BODY.test(p.body || ""); + const ratio = st.total ? st.ai / st.total : 0; + const used = st.ai > 0 || bodyMarker; + return { + number: p.number, title: p.title, branch: p.head && p.head.ref, + author: p.user && p.user.login, + collaborators: p._collab || [], + description: (p.body || "").slice(0, 500), + first_commit_at: (fc && fc.commit && fc.commit.author && fc.commit.author.date) || null, + opened_at: p.created_at || null, + merged_at: p.merged_at || null, + size: det ? { additions: det.additions, deletions: det.deletions, files: det.changed_files } : null, + ai: { + used, + level: !used ? "none" : ratio >= 1 ? "full" : ratio >= 0.5 ? "mostly" : "assisted", + commits_ai: st.ai, commits_total: st.total, body_marker: bodyMarker, + }, + }; + }); + participants = [...people].map(([github, roles]) => ({ github, roles: [...roles] })); + } + + // Participants stay as {github, roles}: mapping to platform users + // happens at REPORT time (lake join vs user/identity metadata) — it + // stays current when mappings change and costs zero API calls here. + + // Deterministic signals + const PATTERNS = { + db_migrations: /(^|\/)(migrations?|db\/migrate|liquibase|flyway)\//i, + auth: /(^|\/)(auth|authn|authz|permissions?|iam)\//i, + infra: /(^|\/)(terraform|infra(structure)?|helm|k8s|\.github\/workflows)\/|(^|\/)Dockerfile$/i, + dependencies: /(^|\/)(package(-lock)?\.json|pnpm-lock\.yaml|yarn\.lock|go\.(mod|sum)|requirements\.txt|Gemfile(\.lock)?|pom\.xml)$/i, + }; + const sensitive_paths = {}; + for (const k of Object.keys(PATTERNS)) sensitive_paths[k] = files.some(f => PATTERNS[k].test(f.filename)); + const signals = { + lines_added: files.reduce((n, f) => n + (f.additions || 0), 0), + lines_deleted: files.reduce((n, f) => n + (f.deletions || 0), 0), + files_changed: files.length, + commits: commits.length, + prs: prs.map(p => p.number), + branches: new Set(prs.map(p => p.branch).filter(Boolean)).size, + intermediate_releases: releases_between, + sensitive_paths, + }; + + let criticality = "critical"; + try { + const gmeta = await np(`/metadata/application/${application_id}`); + if (gmeta.governance && gmeta.governance.criticality) criticality = String(gmeta.governance.criticality); + } catch (e) { /* unclassified -> critical */ } + + let deployment_id = $item.deployment_id; + if (!deployment_id) { + const cand = (deps.results || []).find(d => String(d.release_id) === String(release_id)); + deployment_id = cand && cand.id; + } + + return { + deployment_id, + criticality, + from_release: from ? { id: from.id, semver: from.semver, commit_sha: from.commit_sha } : null, + to_release: { id: to.id, semver: to.semver, commit_sha: to.commit_sha }, + previously_deployed, + releases_between, + commits, files, prs, participants, signals, + ai_usage: { + prs_ai: prs.filter(p => p.ai && p.ai.used).length, + prs_total: prs.length, + commits_ai: prs.reduce((n, p) => n + ((p.ai && p.ai.commits_ai) || 0), 0), + commits_total: prs.reduce((n, p) => n + ((p.ai && p.ai.commits_total) || 0), 0), + }, + }; + errorHandling: + fallbackStep: resolve_failure + + - id: progress_gathered + type: module + pluginType: np-checklist-item-progress + name: "Heartbeat: diff computed" + inputs: + callbackUrl: "${{ steps.trigger.outputs.callbackUrl }}" + callbackToken: "${{ steps.trigger.outputs.callbackToken }}" + message: "Diff computed: ${{ steps.gather.outputs.signals.commits }} commits, ${{ steps.gather.outputs.signals.files_changed }} files, ${{ steps.gather.outputs.releases_between }} intermediate releases. Scoring risk…" + + - id: risk_agent + type: module + pluginType: claude-code-agent + name: "Score change risk" + config: + model: claude-sonnet-4-5 + maxIterations: 3 + systemPrompt: | + You assess deployment risk for nullplatform's own production deploys. + Rules of thumb: config/copy tweaks and small isolated fixes are low; + new features, schema or dependency changes are medium; anything + touching auth, data migrations, infra, or very large / multi-branch + accumulations is high. Be concise and concrete: name the riskiest + part of the change in the rationale. + + The summary must be FUNCTIONAL: describe what this deploy changes in + behavior or capability (features shipped, bugs fixed, what users or + operators will notice), NOT which files were touched — file and line + detail is already shown as structured data next to your summary. + Name a code component only when it is essential to understand the risk. + userPrompt: | + Assess this deployment change. + + Signals: ${{ steps.gather.outputs.signals }} + Commit messages: ${{ steps.gather.outputs.commits }} + PR titles: ${{ steps.gather.outputs.prs }} + Files: ${{ steps.gather.outputs.files }} + + Produce: + - risk: low | medium | high + - rationale: ≤50 words, name the riskiest concrete element + - summary_md: ≤120-word markdown summary of what this deploy changes + FUNCTIONALLY (features, fixes, behavior), for the human who approves + it. Do not list files or line counts. + - short_summary: ONE plain sentence (max 15 words, no markdown) naming + the essential functional change — used where UI space is tight. + - pr_summaries: for EACH PR in the input, a one-line functional summary + (≤25 words) of what that PR ships. + outputSchema: + type: object + required: [risk, rationale, summary_md, short_summary] + properties: + risk: { type: string, enum: [low, medium, high] } + rationale: { type: string } + summary_md: { type: string } + short_summary: { type: string } + pr_summaries: + type: array + items: + type: object + required: [number, summary] + properties: + number: { type: integer } + summary: { type: string } + errorHandling: + fallbackStep: assemble_heuristic + + # LLM path: floors + assemble the final change object. + - id: assemble + type: module + pluginType: code-exec + name: "Apply floors + assemble metadata" + inputs: + change: "${{ steps.gather.outputs }}" + llm: "${{ steps.risk_agent.outputs }}" + config: + code: | + const change = $item.change; + const llm = $item.llm || {}; + const ORDER = { low: 0, medium: 1, high: 2 }; + const floors = []; + const sp = (change.signals && change.signals.sensitive_paths) || {}; + if (sp.db_migrations) floors.push("db_migration"); + if (sp.auth) floors.push("auth_change"); + if (!change.from_release) floors.push("first_deploy"); + const heuristic = (() => { + const s = change.signals || {}; + let n = 0; + if ((s.lines_added || 0) + (s.lines_deleted || 0) > 500) n++; + if ((s.files_changed || 0) > 20) n++; + if ((s.intermediate_releases || 0) > 2) n++; + if (sp.dependencies) n++; + if (sp.infra) n++; + if (sp.db_migrations || sp.auth) n += 2; + return n >= 3 ? "high" : n >= 1 ? "medium" : "low"; + })(); + const scored = llm.risk || heuristic; + const floored = floors.length ? "medium" : "low"; + const risk = ORDER[scored] >= ORDER[floored] ? scored : floored; + const summary_md = llm.summary_md || + `${(change.commits || []).length} commits, ${(change.signals || {}).files_changed || 0} files: ` + + ((change.prs || []).map(p => p.title).join("; ") || (change.commits || []).slice(0, 5).map(c => c.message).join("; ") || "no commit data"); + const rationale = llm.rationale || "Heuristic score (LLM unavailable)."; + const short_summary = llm.short_summary || + (summary_md.split(/[.\n]/)[0] || "").replace(/[*_#`]/g, "").trim().slice(0, 120); + const RANK = { mission_critical: 1, critical: 2, important: 3, standard: 4, internal: 5 }; + const critName = change.criticality || "critical"; + const rank = RANK[critName] || 2; + let mode; + if (change.previously_deployed) mode = "fastpath"; + else if (risk === "low") mode = "auto"; + else if (risk === "medium") mode = rank <= 2 ? "par" : "auto"; + else mode = rank === 1 ? "grupo" : rank <= 3 ? "par" : "auto"; + const auto = mode === "auto" || mode === "fastpath"; + const risk_gate_status = auto ? "passed" : "failed"; + const risk_gate_message = auto + ? (mode === "fastpath" ? "Release already deployed to this scope recently - rollback fastpath" : "Auto-approved: risk " + risk + " x criticality " + critName) + : "Matrix requires " + mode + " approval (risk " + risk + " x criticality " + critName + ") - falls back to manual review until Slack approvals land"; + const participantsLine = (change.participants || []) + .map(p => `${p.github} [${p.roles.join(", ")}]`).join(", ") || "none detected"; + const prSummaries = new Map(((llm && llm.pr_summaries) || []).map(p => [p.number, p.summary])); + const prs_enriched = (change.prs || []).map(p => Object.assign({}, p, { + summary: prSummaries.get(p.number) || (p.description ? p.description.split("\n")[0].slice(0, 160) : null), + })); + const prLines = prs_enriched.map(p => { + const others = (p.collaborators || []).filter(c => c.github !== p.author) + .map(c => `${c.github} (${c.roles.join("/")})`).join(", "); + const aiTag = p.ai && p.ai.used ? ` · 🤖 AI ${p.ai.level}` : ""; + return `- **#${p.number} ${p.title}** — ${p.author}${others ? " · con " + others : ""}${aiTag}${p.summary ? `\n ${p.summary}` : ""}`; + }).join("\n"); + const markdown = [ + `### Change analysis — risk: **${risk.toUpperCase()}**${floors.length ? ` _(floors: ${floors.join(", ")})_` : ""}`, + "", + change.previously_deployed ? "♻️ **This release was already deployed to this scope recently (rollback fastpath candidate).**\n" : "", + summary_md, + prLines ? "\n**PRs:**\n" + prLines : "", + "", + `**From → to:** ${change.from_release ? change.from_release.semver : "(first deploy)"} → ${change.to_release.semver} (${change.releases_between} releases in between)`, + `**Size:** ${change.signals.lines_added}+/${change.signals.lines_deleted}- across ${change.signals.files_changed} files, ${change.signals.commits} commits, ${(change.prs || []).length} PRs`, + `**Participants:** ${participantsLine}`, + `**Why:** ${rationale}`, + ].join("\n"); + const gate_results = [ + { item: "already_deployed", + status: change.previously_deployed ? "passed" : "failed", + message: change.previously_deployed + ? "Release deployed to this scope within the last 10 finalized deploys - rollback fastpath" + : "Release not among the last 10 finalized deploys of this scope" }, + { item: "risk_matrix", status: auto ? "passed" : "failed", message: risk_gate_message }, + { item: "risk_gate", status: risk_gate_status, message: risk_gate_message }, + ]; + return { + metadata: Object.assign({}, change, { + prs: prs_enriched, + risk, risk_rationale: rationale, risk_floors_applied: floors, summary_md, short_summary, + approval: { mode, criticality: critName }, + }), + markdown: markdown + "\n**Decision:** " + mode, + risk, gate_results, decision_mode: mode, + }; + + # Heuristic path (LLM failed): dedicated node so joins stay single-predecessor. + - id: assemble_heuristic + type: module + pluginType: code-exec + name: "Assemble metadata (heuristic, LLM failed)" + inputs: + change: "${{ steps.gather.outputs }}" + config: + code: | + const change = $item.change; + const ORDER = { low: 0, medium: 1, high: 2 }; + const floors = []; + const sp = (change.signals && change.signals.sensitive_paths) || {}; + if (sp.db_migrations) floors.push("db_migration"); + if (sp.auth) floors.push("auth_change"); + if (!change.from_release) floors.push("first_deploy"); + const s = change.signals || {}; + let n = 0; + if ((s.lines_added || 0) + (s.lines_deleted || 0) > 500) n++; + if ((s.files_changed || 0) > 20) n++; + if ((s.intermediate_releases || 0) > 2) n++; + if (sp.dependencies) n++; + if (sp.infra) n++; + if (sp.db_migrations || sp.auth) n += 2; + const scored = n >= 3 ? "high" : n >= 1 ? "medium" : "low"; + const floored = floors.length ? "medium" : "low"; + const risk = ORDER[scored] >= ORDER[floored] ? scored : floored; + const RANK = { mission_critical: 1, critical: 2, important: 3, standard: 4, internal: 5 }; + const critName = change.criticality || "critical"; + const rank = RANK[critName] || 2; + let mode; + if (change.previously_deployed) mode = "fastpath"; + else if (risk === "low") mode = "auto"; + else if (risk === "medium") mode = rank <= 2 ? "par" : "auto"; + else mode = rank === 1 ? "grupo" : rank <= 3 ? "par" : "auto"; + const auto = mode === "auto" || mode === "fastpath"; + const risk_gate_status = auto ? "passed" : "failed"; + const risk_gate_message = auto + ? (mode === "fastpath" ? "Release already deployed to this scope recently - rollback fastpath" : "Auto-approved: risk " + risk + " x criticality " + critName) + : "Matrix requires " + mode + " approval (risk " + risk + " x criticality " + critName + ") - falls back to manual review until Slack approvals land"; + const rationale = "Heuristic score (LLM step failed): size, accumulation and sensitive-path signals only."; + const summary_md = `${(change.commits || []).length} commits, ${s.files_changed || 0} files: ` + + ((change.prs || []).map(p => p.title).join("; ") || "no commit data"); + const short_summary = ((change.prs || [])[0] ? (change.prs || [])[0].title : `${(change.commits || []).length} commits`).slice(0, 120); + const markdown = `### Change analysis — risk: **${risk.toUpperCase()}** _(heuristic)_\n\n${summary_md}\n\n**Why:** ${rationale}`; + const gate_results = [ + { item: "already_deployed", + status: change.previously_deployed ? "passed" : "failed", + message: change.previously_deployed + ? "Release deployed to this scope within the last 10 finalized deploys - rollback fastpath" + : "Release not among the last 10 finalized deploys of this scope" }, + { item: "risk_matrix", status: auto ? "passed" : "failed", message: risk_gate_message }, + { item: "risk_gate", status: risk_gate_status, message: risk_gate_message }, + ]; + return { + metadata: Object.assign({}, change, { + risk, risk_rationale: rationale, risk_floors_applied: floors, summary_md, short_summary, + approval: { mode, criticality: critName }, + }), + markdown: markdown + "\n**Decision:** " + mode, + risk, gate_results, decision_mode: mode, + }; + + # ── Success tail (LLM path) ──────────────────────────────────────────── + - id: write_metadata + type: module + pluginType: code-exec + name: "Persist deployment.metadata.change (upsert)" + inputs: + deployment_id: "${{ steps.gather.outputs.deployment_id }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + metadata: "${{ steps.assemble.outputs.metadata }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { deployment_id, np_api_key, metadata } = $item; + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const url = `https://api.nullplatform.com/metadata/deployment/${deployment_id}/change`; + const call = (method) => fetch(url, { + method, + headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" }, + body: JSON.stringify(metadata), + }); + let res = await call("POST"); + if (!res.ok) { + const text = await res.text(); + if (text.includes("already exists")) res = await call("PATCH"); + else throw new Error(`metadata POST → ${res.status}: ${text}`); + } + if (!res.ok) throw new Error(`metadata PATCH → ${res.status}: ${await res.text()}`); + return { written: true }; + + + - id: resolve + type: module + pluginType: code-exec + name: "Resolve items (change_analysis + risk_gate)" + inputs: + callbackUrl: "${{ steps.trigger.outputs.callbackUrl }}" + callbackToken: "${{ steps.trigger.outputs.callbackToken }}" + out: "${{ steps.assemble.outputs }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { callbackUrl, callbackToken, out } = $item; + const H = { authorization: `Bearer ${callbackToken}`, "content-type": "application/json" }; + const post = (url, body) => fetch(url, { method: "POST", headers: H, body: JSON.stringify(body) }); + const patch = (url, body) => fetch(url, { method: "PATCH", headers: H, body: JSON.stringify(body) }); + await post(`${callbackUrl}/log`, { level: "info", message: `Risk: ${out.risk} - ${out.metadata.risk_rationale}` }).catch(() => null); + await post(`${callbackUrl}/log`, { level: "info", message: `Matrix decision: ${out.decision_mode} (criticality ${out.metadata.criticality || "critical"})` }).catch(() => null); + let r = await patch(callbackUrl, { status: "passed", message: `Risk ${out.risk}: ${out.metadata.short_summary || "change analyzed"}`, details: { markdown: out.markdown } }); + if (!r.ok) throw new Error(`change_analysis resolve -> ${r.status}: ${await r.text()}`); + // Sibling gates (cross_validation group, template v6). Tolerated when + // the item is absent (404), when the token is item-scoped pre-#260 + // (401), or when the run already resolved (409). + const results = {}; + for (const g of out.gate_results || []) { + const url = callbackUrl.replace(/\/items\/[^/]+$/, `/items/${g.item}`); + const rr = await patch(url, { status: g.status, message: g.message }); + results[g.item] = rr.ok ? g.status : `skipped_${rr.status}`; + if (!rr.ok && ![401, 404, 409].includes(rr.status)) throw new Error(`${g.item} resolve -> ${rr.status}: ${await rr.text()}`); + } + return { resolved: true, gates: results }; + + + # ── Heuristic tail (LLM failed; dedicated nodes, no mixed joins) ────── + - id: write_metadata_h + type: module + pluginType: code-exec + name: "Persist deployment.metadata.change (heuristic, upsert)" + inputs: + deployment_id: "${{ steps.gather.outputs.deployment_id }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + metadata: "${{ steps.assemble_heuristic.outputs.metadata }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { deployment_id, np_api_key, metadata } = $item; + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const url = `https://api.nullplatform.com/metadata/deployment/${deployment_id}/change`; + const call = (method) => fetch(url, { + method, + headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" }, + body: JSON.stringify(metadata), + }); + let res = await call("POST"); + if (!res.ok) { + const text = await res.text(); + if (text.includes("already exists")) res = await call("PATCH"); + else throw new Error(`metadata POST → ${res.status}: ${text}`); + } + if (!res.ok) throw new Error(`metadata PATCH → ${res.status}: ${await res.text()}`); + return { written: true }; + + + - id: resolve_h + type: module + pluginType: code-exec + name: "Resolve items (change_analysis + risk_gate)" + inputs: + callbackUrl: "${{ steps.trigger.outputs.callbackUrl }}" + callbackToken: "${{ steps.trigger.outputs.callbackToken }}" + out: "${{ steps.assemble_heuristic.outputs }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { callbackUrl, callbackToken, out } = $item; + const H = { authorization: `Bearer ${callbackToken}`, "content-type": "application/json" }; + const post = (url, body) => fetch(url, { method: "POST", headers: H, body: JSON.stringify(body) }); + const patch = (url, body) => fetch(url, { method: "PATCH", headers: H, body: JSON.stringify(body) }); + await post(`${callbackUrl}/log`, { level: "info", message: `Risk: ${out.risk} - ${out.metadata.risk_rationale}` }).catch(() => null); + await post(`${callbackUrl}/log`, { level: "info", message: `Matrix decision: ${out.decision_mode} (criticality ${out.metadata.criticality || "critical"})` }).catch(() => null); + let r = await patch(callbackUrl, { status: "passed", message: `Risk ${out.risk}: ${out.metadata.short_summary || "change analyzed"}`, details: { markdown: out.markdown } }); + if (!r.ok) throw new Error(`change_analysis resolve -> ${r.status}: ${await r.text()}`); + // Sibling gates (cross_validation group, template v6). Tolerated when + // the item is absent (404), when the token is item-scoped pre-#260 + // (401), or when the run already resolved (409). + const results = {}; + for (const g of out.gate_results || []) { + const url = callbackUrl.replace(/\/items\/[^/]+$/, `/items/${g.item}`); + const rr = await patch(url, { status: g.status, message: g.message }); + results[g.item] = rr.ok ? g.status : `skipped_${rr.status}`; + if (!rr.ok && ![401, 404, 409].includes(rr.status)) throw new Error(`${g.item} resolve -> ${rr.status}: ${await rr.text()}`); + } + return { resolved: true, gates: results }; + + + # Reached only via gather's `error_handling.fallback_step`: the analysis + # itself crashed (NP/GitHub unavailable, bad repo, etc.). The item is + # informational so this does not block the deploy; the error is surfaced + # verbatim for debugging. + - id: resolve_failure + type: module + pluginType: np-checklist-item-resolve + name: "Resolve as failed (analysis crashed)" + inputs: + callbackUrl: "${{ steps.trigger.outputs.callbackUrl }}" + callbackToken: "${{ steps.trigger.outputs.callbackToken }}" + status: failed + message: "Change analysis crashed: ${{ steps.gather.error.message }}" + details: + markdown: "The analysis workflow failed before producing a result. Deploy is NOT blocked (informational item). Error: ${{ steps.gather.error.message }}" + +connections: + - { id: c1, from: trigger, to: progress } + - { id: c2, from: progress, to: gather } + - { id: c3, from: gather, to: progress_gathered } + - { id: c3b, from: progress_gathered, to: risk_agent } + - { id: c4, from: risk_agent, to: assemble } + - { id: c5, from: assemble, to: resolve } + - { id: c6, from: resolve, to: write_metadata } + # Dormant edges: keep the fallback tails out of the entry-point set; + # the engine routes to them on terminal failure via fallback_step. + - { id: c7, from: gather, to: resolve_failure, condition: "false" } + - { id: c8, from: risk_agent, to: assemble_heuristic, condition: "false" } + - { id: c9, from: assemble_heuristic, to: resolve_h } + - { id: c10, from: resolve_h, to: write_metadata_h } diff --git a/deploy-governance/deploy-weekly-summary.yaml b/deploy-governance/deploy-weekly-summary.yaml new file mode 100644 index 0000000..dd1e294 --- /dev/null +++ b/deploy-governance/deploy-weekly-summary.yaml @@ -0,0 +1,415 @@ +# Deploy Weekly Summary — aggregates the week's production deploys into +# functional summaries at APP and NAMESPACE level, persisted as rolling +# metadata (last 53 weeks) for reporting/dashboards. +# +# Data source: deployment.metadata.change (written by deploy-change-analysis) +# — the per-deploy summary_md and PR summaries already exist, so this is pure +# text aggregation: NO GitHub calls, NO re-analysis. One LLM call per week. +# +# Triggers: +# - cron: Mondays 07:00 America/Argentina/Buenos_Aires (summarizes the week +# that just ended: Mon 00:00 UTC → next Mon 00:00 UTC). +# - webhook (manual/testing): POST body {"week_offset": 1} +# week_offset=1 → last full week; 2 → the week before; default 1. +# +# Storage: metadata `deploy_summaries` on application and namespace entities. +# Rolling window: newest-first array, trimmed to 53 entries (~1 year). +# Idempotent: re-running a week replaces that week's entry (upsert by +# week_start), so manual re-fires are safe. +id: deploy-weekly-summary +key: deploy-weekly-summary +name: "Deploy Weekly Summary (app + namespace)" +description: > + Weekly cron: collects the previous week's production deployments and their + change metadata, produces functional summaries per application and per + namespace with one LLM call, and persists them as rolling 53-week metadata + (deploy_summaries) on each entity. +semantic_version: 1.0.0 +path: "/checklist/deploy" + +inputs: + triggerPayload: + type: object + default: {} + +steps: + # 09:00 keeps it well clear of the 01:30 nightly backfill: by the time the + # weekly runs, every deploy of the closing week already has its metadata. + - id: trigger + type: trigger + pluginType: cron + name: "Weekly (Mondays 09:00 AR)" + config: + schedule: "0 9 * * 1" + timezone: "America/Argentina/Buenos_Aires" + + - id: gather + type: module + pluginType: code-exec + name: "Collect week's deploys + change metadata" + inputs: + week_offset: "${{ workflow.inputs.body.week_offset }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const npApiKey = $item.np_api_key; + const weekOffset = Number($item.week_offset) || 1; + + // Previous full week in UTC: Monday 00:00 → next Monday 00:00. + const now = new Date(); + const day = (now.getUTCDay() + 6) % 7; // 0 = Monday + const thisMonday = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate() - day)); + const weekStart = new Date(thisMonday.getTime() - weekOffset * 7 * 86400000); + const weekEnd = new Date(weekStart.getTime() + 7 * 86400000); + const iso = (d) => d.toISOString().slice(0, 10); + const fmt = (d) => d.toISOString().slice(0, 19).replace("T", " "); + + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: npApiKey }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + + // Lake: finalized production deploys inside the week, with app + namespace. + const sql = ` + WITH prod_scopes AS ( + SELECT DISTINCT scope_id FROM core_entities_scope_dimension + WHERE dimension_slug = 'environment' AND value_slug = 'production' + ) + SELECT d.id, s.application_id, a.app_name, a.namespace_id, n.namespace_name, d.created_at + FROM core_entities_deployment d FINAL + JOIN core_entities_scope s FINAL ON s.id = d.scope_id + JOIN core_entities_application a FINAL ON a.app_id = s.application_id + LEFT JOIN core_entities_namespace n FINAL ON n.namespace_id = a.namespace_id + WHERE d.status = 'finalized' + AND d.scope_id IN (SELECT scope_id FROM prod_scopes) + AND d.created_at >= toDateTime('${fmt(weekStart)}') + AND d.created_at < toDateTime('${fmt(weekEnd)}') + ORDER BY d.created_at`; + const lakeRes = await fetch("https://api.nullplatform.com/data/lake/query", { + method: "POST", + headers: { authorization: `Bearer ${npToken}`, "content-type": "application/json" }, + body: JSON.stringify({ query: sql }), + }); + if (!lakeRes.ok) throw new Error(`lake query → ${lakeRes.status}: ${(await lakeRes.text()).slice(0, 200)}`); + // Headerless TSV, columns in SELECT order; numbers arrive as strings. + const COLS = ["deployment_id", "application_id", "app_name", "namespace_id", "namespace_name", "created_at"]; + const rows = (await lakeRes.text()).trim().split("\n").filter(Boolean).map((line) => { + const parts = line.split("\t"); + const r = {}; + for (let i = 0; i < COLS.length; i++) r[COLS[i]] = parts[i]; + return r; + }); + + // Fetch change metadata per deploy (already computed by the analysis + // workflow; 404 = deploy without analysis, still counted). + const deploys = []; + for (const r of rows) { + let change = null; + try { + const res = await fetch(`https://api.nullplatform.com/metadata/deployment/${r.deployment_id}/change`, { + headers: { authorization: `Bearer ${npToken}` }, + }); + if (res.ok) change = await res.json(); + } catch (e) { /* keep null */ } + deploys.push({ + deployment_id: r.deployment_id, + application_id: r.application_id, + app_name: r.app_name, + namespace_id: r.namespace_id, + namespace_name: r.namespace_name || "(sin namespace)", + date: r.created_at, + risk: change && change.risk || null, + mode: change && change.approval && change.approval.mode || null, + fastpath: !!(change && change.previously_deployed), + from_semver: change && change.from_release && change.from_release.semver || null, + to_semver: change && change.to_release && change.to_release.semver || null, + summary_md: change && change.summary_md || null, + floors: (change && change.risk_floors_applied) || [], + prs: ((change && change.prs) || []).map(p => ({ + number: p.number, title: p.title, author: p.author, summary: p.summary || null, + ai: (p.ai && p.ai.level) || null, + collaborators: (p.collaborators || []).map(c => ({ github: c.github, roles: c.roles })), + })), + participants: ((change && change.participants) || []).map(p => ({ github: p.github, roles: p.roles || [] })), + }); + } + + // Group by app; count risk histogram. + const byApp = new Map(); + for (const d of deploys) { + if (!byApp.has(d.application_id)) byApp.set(d.application_id, { + application_id: d.application_id, app_name: d.app_name, + namespace_id: d.namespace_id, namespace_name: d.namespace_name, + deploys: [], + }); + byApp.get(d.application_id).deploys.push(d); + } + const hist = (list) => { + const h = { low: 0, medium: 0, high: 0, unknown: 0 }; + for (const d of list) h[d.risk || "unknown"]++; + return h; + }; + const mergeRoles = (deploys) => { + const m = new Map(); + for (const d of deploys) for (const p of d.participants || []) { + if (!m.has(p.github)) m.set(p.github, new Set()); + for (const r of p.roles) m.get(p.github).add(r); + } + return [...m].map(([github, roles]) => ({ github, roles: [...roles] })); + }; + const mergePrs = (deploys) => { + const m = new Map(); + for (const d of deploys) for (const p of d.prs || []) if (!m.has(p.number)) m.set(p.number, p); + return [...m.values()]; + }; + const apps = [...byApp.values()].map(a => Object.assign(a, { + people: mergeRoles(a.deploys), + all_prs: mergePrs(a.deploys), + stats: { + deploys: a.deploys.length, + risk: hist(a.deploys), + fastpath: a.deploys.filter(d => d.fastpath).length, + db_migrations: a.deploys.filter(d => (d.floors || []).includes("db_migration")).length, + ai_prs: mergePrs(a.deploys).filter(p => p.ai && p.ai !== "none").length, + prs: mergePrs(a.deploys).length, + participants: mergeRoles(a.deploys).length, + }, + })); + + const byNs = new Map(); + for (const a of apps) { + if (!byNs.has(a.namespace_id)) byNs.set(a.namespace_id, { + namespace_id: a.namespace_id, namespace_name: a.namespace_name, apps: [], + }); + byNs.get(a.namespace_id).apps.push(a.application_id); + } + const namespaces = [...byNs.values()]; + + // Compact view for the LLM prompt: the full `apps` (with people, + // collaborators, etc.) blows past the agent runner's prompt-size + // limit on busy weeks (~180KB → exit 126). The narrative does not + // name people, so the LLM only needs summaries + PR titles. + const llm_view = apps.map(a => ({ + application_id: a.application_id, app_name: a.app_name, + namespace_id: a.namespace_id, namespace_name: a.namespace_name, + deploys: a.deploys.map(d => ({ + date: String(d.date || "").slice(0, 10), + risk: d.risk, mode: d.mode, fastpath: d.fastpath, + from: d.from_semver, to: d.to_semver, + summary: d.summary_md ? String(d.summary_md).slice(0, 700) : null, + prs: (d.prs || []).slice(0, 20).map(p => ({ + n: p.number, t: p.title, + s: p.summary ? String(p.summary).slice(0, 180) : null, + })), + })), + })); + + return { + week_start: iso(weekStart), week_end: iso(new Date(weekEnd.getTime() - 86400000)), + has_data: apps.length > 0, + total_deploys: deploys.length, + apps, namespaces, llm_view, + }; + + - id: summarize + type: module + pluginType: claude-code-agent + name: "Weekly functional summaries (app + namespace)" + config: + model: claude-opus-4-8 + maxIterations: 15 + systemPrompt: | + IMPORTANT: All the data you need is in the user message. Do NOT use + tools (no Read, no Write, no exploration) — produce the structured + output directly from the input, in your first response. + + You write weekly release-notes-style summaries of what shipped to + production at nullplatform, from per-deploy summaries that already + exist. Summaries must be FUNCTIONAL: features shipped, bugs fixed, + behavior changes — what users or operators will notice. Never list + files, deploy counts alone, or version numbers without meaning. + Merge redundant deploys of the same change (rollback+refix, repeated + fastpaths) into a single narrative line. Be concrete and concise. + userPrompt: | + Week ${{ steps.gather.outputs.week_start }} → ${{ steps.gather.outputs.week_end }}. + Production deploys grouped by application. Per deploy: date, risk, + approval mode, fastpath (rollback/re-deploy), from→to version, + functional summary, and prs (n = PR number, t = title, s = summary): + + ${{ steps.gather.outputs.llm_view }} + + Namespaces (grouping of the apps above): + + ${{ steps.gather.outputs.namespaces }} + + Produce: + - app_summaries: for EACH application in the input, a markdown summary + (≤180 words) of what shipped to production that week, functionally, + PLUS short_summary: up to 3 markdown bullets (max 10 words each) + with the week's highlights for that app — for tight UI spaces. + Reference the key PRs by number (e.g. "(#123)"). Do NOT name people + in the narrative — participants and roles are stored as structured + data next to the summary. If an app's deploys have no analysis data + (summary_md null), say what can be inferred from versions and note + the analysis is missing. + - namespace_summaries: for EACH namespace, a markdown roll-up (≤250 + words) built FROM the app summaries, PLUS short_summary: up to 3 + markdown bullets (max 10 words each) with the namespace's week + highlights. The roll-up covers: the week's theme, notable + features/fixes across its apps (reference the most relevant PRs by + number and app), and anything risky (high-risk deploys, db + migrations, rollbacks). Do NOT name people in the narrative. + outputSchema: + type: object + required: [app_summaries, namespace_summaries] + properties: + app_summaries: + type: array + items: + type: object + required: [application_id, summary_md, short_summary] + properties: + application_id: { type: string } + summary_md: { type: string } + short_summary: + type: array + maxItems: 3 + items: { type: string } + namespace_summaries: + type: array + items: + type: object + required: [namespace_id, summary_md, short_summary] + properties: + namespace_id: { type: string } + summary_md: { type: string } + short_summary: + type: array + maxItems: 3 + items: { type: string } + + - id: write + type: module + pluginType: code-exec + name: "Persist rolling deploy_summaries (app + namespace)" + inputs: + gathered: "${{ steps.gather.outputs }}" + llm: "${{ steps.summarize.outputs }}" + np_api_key: "${{ secrets.NP_API_KEY }}" + config: + network: + allowedHosts: ["api.nullplatform.com"] + code: | + const { gathered, np_api_key } = $item; + const llm = $item.llm || {}; + const MAX_WEEKS = 53; // ~1 year rolling window + const tokRes = await fetch("https://api.nullplatform.com/token", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ api_key: np_api_key }), + }); + if (!tokRes.ok) throw new Error(`NP token exchange failed: ${tokRes.status}`); + const npToken = (await tokRes.json()).access_token; + const H = { authorization: `Bearer ${npToken}`, "content-type": "application/json" }; + + const appSum = new Map((llm.app_summaries || []).map(s => [String(s.application_id), s.summary_md])); + const nsSum = new Map((llm.namespace_summaries || []).map(s => [String(s.namespace_id), s.summary_md])); + const appShort = new Map((llm.app_summaries || []).map(s => [String(s.application_id), s.short_summary])); + const nsShort = new Map((llm.namespace_summaries || []).map(s => [String(s.namespace_id), s.short_summary])); + const derivedShort = (md) => [(String(md || "").split(/[.\n]/)[0] || "").replace(/[*_#`]/g, "").trim().slice(0, 100)]; + + // Fallback narrative when the LLM output is missing for an entity. + const fallback = (deploys) => deploys.map(d => + `- ${d.to_semver || d.deployment_id}: ${d.summary_md ? d.summary_md.split("\n")[0] : "(sin análisis)"}` + ).join("\n"); + + const upsert = async (entity, id, entry) => { + const url = `https://api.nullplatform.com/metadata/${entity}/${id}/deploy_summaries`; + let current = { weeks: [] }; + const getRes = await fetch(url, { headers: H }); + if (getRes.ok) { + try { const j = await getRes.json(); if (j && Array.isArray(j.weeks)) current = j; } catch (e) {} + } + // Idempotent by week_start: replace if re-run; newest first; trim to a year. + const weeks = [entry, ...current.weeks.filter(w => w.week_start !== entry.week_start)] + .sort((a, b) => (a.week_start < b.week_start ? 1 : -1)) + .slice(0, MAX_WEEKS); + // latest_summary mirrors the newest week: it is what the Catalog + // card shows (weeks stays for the detail view). + const latest = weeks[0]; + const body = JSON.stringify({ + latest_summary: `**Semana ${latest.week_start} → ${latest.week_end}** — ${latest.deploys} deploys (risk: ${latest.risk.low || 0} low / ${latest.risk.medium || 0} medium / ${latest.risk.high || 0} high)\n\n${latest.summary_md}`, + latest_short: latest.short_summary || derivedShort(latest.summary_md), + weeks, + }); + let res = await fetch(url, { method: "POST", headers: H, body }); + if (!res.ok) { + const text = await res.text(); + if (text.includes("already exists")) res = await fetch(url, { method: "PATCH", headers: H, body }); + else throw new Error(`metadata POST ${entity}/${id} → ${res.status}: ${text.slice(0, 200)}`); + } + if (!res.ok) throw new Error(`metadata PATCH ${entity}/${id} → ${res.status}`); + }; + + const generated_at = new Date().toISOString(); + let appsWritten = 0, nsWritten = 0; + + for (const a of gathered.apps || []) { + await upsert("application", a.application_id, { + week_start: gathered.week_start, week_end: gathered.week_end, + deploys: a.stats.deploys, risk: a.stats.risk, fastpath: a.stats.fastpath, + db_migrations: a.stats.db_migrations, ai_prs: a.stats.ai_prs, + summary_md: appSum.get(String(a.application_id)) || fallback(a.deploys), + short_summary: appShort.get(String(a.application_id)) || derivedShort(appSum.get(String(a.application_id)) || fallback(a.deploys)), + prs: (a.all_prs || []).map(p => ({ number: p.number, title: p.title, author: p.author, summary: p.summary, ai: p.ai || null })), + people: a.people || [], + generated_at, + }); + appsWritten++; + } + + for (const ns of gathered.namespaces || []) { + const nsApps = (gathered.apps || []).filter(a => String(a.namespace_id) === String(ns.namespace_id)); + const agg = { low: 0, medium: 0, high: 0, unknown: 0 }; + for (const a of nsApps) for (const k of Object.keys(agg)) agg[k] += a.stats.risk[k] || 0; + // People across the namespace, with PR counts for "top contributors". + const prCount = new Map(); + const roleMap = new Map(); + for (const a of nsApps) { + for (const p of a.all_prs || []) prCount.set(p.author, (prCount.get(p.author) || 0) + 1); + for (const per of a.people || []) { + if (!roleMap.has(per.github)) roleMap.set(per.github, new Set()); + for (const r of per.roles) roleMap.get(per.github).add(r); + } + } + const contributors = [...roleMap].map(([github, roles]) => ({ + github, roles: [...roles], prs: prCount.get(github) || 0, + })).sort((x, y) => y.prs - x.prs); + await upsert("namespace", ns.namespace_id, { + week_start: gathered.week_start, week_end: gathered.week_end, + deploys: nsApps.reduce((n, a) => n + a.stats.deploys, 0), + apps: nsApps.map(a => ({ id: a.application_id, name: a.app_name, deploys: a.stats.deploys })), + risk: agg, + prs: nsApps.reduce((n, a) => n + (a.all_prs || []).length, 0), + ai_prs: nsApps.reduce((n, a) => n + (a.stats.ai_prs || 0), 0), + db_migrations: nsApps.reduce((n, a) => n + a.stats.db_migrations, 0), + contributors, + summary_md: nsSum.get(String(ns.namespace_id)) || + nsApps.map(a => `**${a.app_name}**: ${(appSum.get(String(a.application_id)) || "(sin resumen)").split("\n")[0]}`).join("\n"), + short_summary: nsShort.get(String(ns.namespace_id)) || derivedShort(nsSum.get(String(ns.namespace_id)) || ""), + generated_at, + }); + nsWritten++; + } + + return { week_start: gathered.week_start, apps_written: appsWritten, namespaces_written: nsWritten, total_deploys: gathered.total_deploys }; + +connections: + - { id: c1, from: trigger, to: gather } + - { id: c2, from: gather, to: summarize } + - { id: c3, from: summarize, to: write } diff --git a/deploy-governance/setup/01-catalog-specs.sh b/deploy-governance/setup/01-catalog-specs.sh new file mode 100644 index 0000000..ce1faa2 --- /dev/null +++ b/deploy-governance/setup/01-catalog-specs.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# One-time catalog setup for the deploy-governance suite. +# +# Creates the metadata specifications the workflows read and write: +# application/governance — criticality (named enum, UI-selectable) +# user/identity — github_username (GitHub -> NP user mapping) +# deployment/change — per-deploy change analysis (full schema, +# sourced from ../specs/deployment-change.spec.json) +# application/deploy_summaries — weekly rolling summaries (app level) +# namespace/deploy_summaries — weekly rolling summaries (namespace level) +# +# Usage: +# NP_API_KEY=... NP_ORGANIZATION_ID= ./01-catalog-specs.sh +# +# Idempotent-ish: re-running when a spec already exists prints the API error +# for that spec and continues (specs are keyed by entity+metadata per NRN). +# Exception: deployment/change is a true upsert — when it already exists its +# schema+description are PATCHed in place, so schema evolution ships by +# editing specs/deployment-change.spec.json and re-running this script. +set -euo pipefail + +: "${NP_API_KEY:?set NP_API_KEY}" +: "${NP_ORGANIZATION_ID:?set NP_ORGANIZATION_ID}" +API="https://api.nullplatform.com" +NRN="organization=${NP_ORGANIZATION_ID}" + +TOKEN=$(curl -sf -X POST "$API/token" -H 'content-type: application/json' \ + -d "{\"api_key\":\"$NP_API_KEY\"}" | jq -r .access_token) + +create_spec() { + local body="$1" name + name=$(echo "$body" | jq -r '.entity + "/" + .metadata') + echo "== $name" + curl -s -X POST "$API/metadata/metadata_specification" \ + -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \ + -d "$body" | jq -c '{id: (.id // null), error: (.message // null)}' +} + +# --- application/governance: criticality as a NAMED enum (renders as a +# select with explanations in the catalog UI via oneOf + title). +create_spec "$(jq -n --arg nrn "$NRN" '{ + nrn: $nrn, entity: "application", metadata: "governance", + name: "Application governance", + description: "Governance attributes: criticality drives the deploy risk matrix.", + schema: { + type: "object", additionalProperties: true, + properties: { + criticality: { + type: "string", + description: "How critical this application is for the business", + oneOf: [ + { const: "mission_critical", title: "Mission critical - outage stops the business" }, + { const: "critical", title: "Critical - customer-facing impact within minutes" }, + { const: "important", title: "Important - degrades operations, no immediate customer impact" }, + { const: "standard", title: "Standard - internal tooling with workarounds" }, + { const: "internal", title: "Internal - non-productive or administrative" } + ], + visibleOn: ["create", "read", "update", "list"] + } + } + } +}')" + +# --- user/identity: GitHub handle mapping for participant extraction. +create_spec "$(jq -n --arg nrn "$NRN" '{ + nrn: $nrn, entity: "user", metadata: "identity", + name: "User identity", + description: "External identities of the user; github_username maps commit/PR authors to platform users.", + schema: { + type: "object", additionalProperties: true, + properties: { github_username: { type: "string" } } + } +}')" + +# --- deployment/change: the per-deploy analysis document. Full schema lives +# in specs/deployment-change.spec.json (single source of truth — the metadata +# service VALIDATES every write against it, so before changing it check that +# real stored docs still validate; see the README "Metadata contract" section). +# Upserted: created if absent, otherwise schema+description PATCHed in place. +SPEC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../specs" && pwd)" +CHANGE_SPEC="$(jq --arg nrn "$NRN" '. + {nrn: $nrn}' "$SPEC_DIR/deployment-change.spec.json")" +echo "== deployment/change" +out=$(curl -s -X POST "$API/metadata/metadata_specification" \ + -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \ + -d "$CHANGE_SPEC") +if [[ "$(jq -r '.id // empty' <<<"$out")" != "" ]]; then + jq -c '{id: .id}' <<<"$out" +else + sid=$(curl -s "$API/metadata/metadata_specification?nrn=$NRN&limit=100" \ + -H "authorization: Bearer $TOKEN" \ + | jq -r '(.results // .) | map(select(.entity=="deployment" and .metadata=="change")) | .[0].id // empty') + if [[ -z "$sid" ]]; then + echo "FAILED: could not create nor find deployment/change: $(jq -c '{error: .message}' <<<"$out")"; exit 1 + fi + curl -s -X PATCH "$API/metadata/metadata_specification/$sid" \ + -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \ + -d "$(jq -c '{schema, description}' <<<"$CHANGE_SPEC")" \ + | jq -c '{id: (.id // null), error: (.message // null)}' +fi + +# --- deploy_summaries (app + namespace): rolling weekly summaries. The UI +# card renders only latest_summary (visibleOn); weeks is API/lake-only data. +weekly_schema() { + local prs_schema="$1" + jq -n --argjson prs "$prs_schema" '{ + type: "object", additionalProperties: true, + properties: { + latest_summary: { + type: "string", + description: "Functional summary of the latest week of production deploys", + visibleOn: ["list", "read"] + }, + weeks: { + type: "array", + description: "Weekly history (rolling 53 weeks, newest first)", + visibleOn: [], + items: { + type: "object", additionalProperties: true, + properties: { + week_start: { type: "string" }, + week_end: { type: "string" }, + deploys: { type: "integer" }, + risk: { type: "object", additionalProperties: true }, + summary_md: { type: "string" }, + prs: $prs, + generated_at: { type: "string" } + } + } + } + } + }' +} + +create_spec "$(jq -n --arg nrn "$NRN" --argjson schema "$(weekly_schema '{ "type": "array", "items": { "type": "object", "additionalProperties": true } }')" '{ + nrn: $nrn, entity: "application", metadata: "deploy_summaries", + name: "Application weekly deploy summaries", + description: "Weekly functional summaries of production deploys (rolling 53 weeks). Written by the deploy-weekly-summary workflow.", + schema: $schema +}')" + +create_spec "$(jq -n --arg nrn "$NRN" --argjson schema "$(weekly_schema '{ "type": "integer" }')" '{ + nrn: $nrn, entity: "namespace", metadata: "deploy_summaries", + name: "Namespace weekly deploy summaries", + description: "Weekly functional roll-ups of production deploys per namespace (rolling 53 weeks). Written by the deploy-weekly-summary workflow.", + schema: $schema +}')" + +echo "Done. Remember: the metadata service VALIDATES writes against these" +echo "schemas — the app spec stores prs as an array of objects, the namespace" +echo "spec as a count. Keep the workflow write step and the specs in sync." diff --git a/deploy-governance/specs/deployment-change.spec.json b/deploy-governance/specs/deployment-change.spec.json new file mode 100644 index 0000000..46d6f23 --- /dev/null +++ b/deploy-governance/specs/deployment-change.spec.json @@ -0,0 +1,455 @@ +{ + "entity": "deployment", + "metadata": "change", + "name": "Deployment change analysis", + "description": "Computed by the deploy-change-analysis workflow (and the categories backfill): diff between deployed and candidate release, participants, signals, AI usage, risk and matrix decision. Machine-written — humans never edit it. visibleOn: [read] → deployment detail only, never in list columns nor create/update forms. Raw payloads (commits, files) are intentionally NOT declared as properties: they stay stored (additionalProperties) but the schema-driven UI does not render them; summary_md narrates them. See design doc 13-deploy-governance.", + "schema": { + "type": "object", + "additionalProperties": true, + "visibleOn": [ + "read" + ], + "properties": { + "risk": { + "type": "string", + "title": "Risk", + "enum": [ + "low", + "medium", + "high" + ], + "description": "Scored by LLM + deterministic floors (db migrations, auth changes, first deploy can never be low)", + "visibleOn": [ + "read" + ] + }, + "short_summary": { + "type": "string", + "title": "Summary", + "description": "One-line functional summary of the change", + "visibleOn": [ + "read" + ] + }, + "summary_md": { + "type": "string", + "title": "Change analysis", + "contentMediaType": "text/markdown", + "description": "Narrative of the change: PRs with authors/collaborators/AI usage, size, participants, risk rationale", + "visibleOn": [ + "read" + ] + }, + "risk_rationale": { + "type": "string", + "title": "Risk rationale", + "visibleOn": [ + "read" + ] + }, + "risk_floors_applied": { + "type": "array", + "title": "Risk floors applied", + "description": "Hard rules that bounded the LLM score (db_migration, auth_change, first_deploy)", + "items": { + "type": "string" + }, + "visibleOn": [ + "read" + ] + }, + "change_categories": { + "type": "array", + "title": "Change categories", + "description": "LLM classification of what the deploy ships, relative to the app's product domain", + "visibleOn": [ + "read" + ], + "items": { + "type": "object", + "additionalProperties": true, + "properties": { + "category": { + "type": "string", + "title": "Category", + "description": "feature | improvement | bugfix | security | performance | refactor | dependencies | infra | config | docs | tests | schema_change | revert" + }, + "count": { + "type": "integer", + "title": "Count" + }, + "notes": { + "type": "string", + "title": "Notes" + } + } + } + }, + "breaking": { + "type": "boolean", + "title": "Breaking change", + "visibleOn": [ + "read" + ] + }, + "hotfix": { + "type": "boolean", + "title": "Hotfix", + "visibleOn": [ + "read" + ] + }, + "approval": { + "type": "object", + "title": "Matrix decision", + "additionalProperties": true, + "visibleOn": [ + "read" + ], + "properties": { + "mode": { + "type": "string", + "title": "Approval mode", + "description": "auto | fastpath | par | grupo — decided by risk × app criticality" + }, + "criticality": { + "type": "string", + "title": "App criticality" + } + } + }, + "criticality": { + "type": "string", + "title": "App criticality (at analysis time)", + "visibleOn": [] + }, + "from_release": { + "type": [ + "object", + "null" + ], + "title": "From release", + "description": "Release deployed in the target scope before this deploy; null on first deploy", + "additionalProperties": true, + "visibleOn": [ + "read" + ], + "properties": { + "id": { + "type": [ + "integer", + "string" + ], + "visibleOn": [] + }, + "semver": { + "type": "string", + "title": "Version" + }, + "commit_sha": { + "type": "string", + "title": "Commit" + } + } + }, + "to_release": { + "type": "object", + "title": "To release", + "additionalProperties": true, + "visibleOn": [ + "read" + ], + "properties": { + "id": { + "type": [ + "integer", + "string" + ], + "visibleOn": [] + }, + "semver": { + "type": "string", + "title": "Version" + }, + "commit_sha": { + "type": "string", + "title": "Commit" + } + } + }, + "releases_between": { + "type": "integer", + "title": "Intermediate releases", + "description": "Releases accumulated between from and to", + "visibleOn": [ + "read" + ] + }, + "previously_deployed": { + "type": "boolean", + "title": "Already deployed recently", + "description": "Candidate release is among the last 10 finalized deploys of the scope (rollback fastpath)", + "visibleOn": [ + "read" + ] + }, + "signals": { + "type": "object", + "title": "Signals", + "description": "Deterministic size/shape signals feeding the risk score", + "additionalProperties": true, + "visibleOn": [ + "read" + ], + "properties": { + "lines_added": { + "type": "integer", + "title": "Lines added" + }, + "lines_deleted": { + "type": "integer", + "title": "Lines deleted" + }, + "files_changed": { + "type": "integer", + "title": "Files changed" + }, + "commits": { + "type": "integer", + "title": "Commits" + }, + "branches": { + "type": "integer", + "title": "Branches" + }, + "intermediate_releases": { + "type": "integer", + "title": "Intermediate releases", + "visibleOn": [] + }, + "prs": { + "type": "array", + "items": { + "type": "integer" + }, + "title": "PR numbers", + "visibleOn": [] + }, + "sensitive_paths": { + "type": "object", + "title": "Sensitive paths touched", + "additionalProperties": true, + "properties": { + "db_migrations": { + "type": "boolean", + "title": "DB migrations" + }, + "auth": { + "type": "boolean", + "title": "Auth" + }, + "infra": { + "type": "boolean", + "title": "Infra" + }, + "dependencies": { + "type": "boolean", + "title": "Dependencies" + } + } + } + } + }, + "prs": { + "type": "array", + "title": "Pull requests", + "visibleOn": [ + "read" + ], + "items": { + "type": "object", + "additionalProperties": true, + "properties": { + "number": { + "type": "integer", + "title": "PR #" + }, + "title": { + "type": "string", + "title": "Title" + }, + "author": { + "type": "string", + "title": "Author" + }, + "summary": { + "type": [ + "string", + "null" + ], + "title": "Summary" + }, + "branch": { + "type": "string", + "title": "Branch", + "visibleOn": [] + }, + "description": { + "type": "string", + "visibleOn": [] + }, + "opened_at": { + "type": [ + "string", + "null" + ], + "visibleOn": [] + }, + "first_commit_at": { + "type": [ + "string", + "null" + ], + "visibleOn": [] + }, + "merged_at": { + "type": [ + "string", + "null" + ], + "title": "Merged at" + }, + "size": { + "type": [ + "object", + "null" + ], + "title": "Size", + "additionalProperties": true, + "properties": { + "additions": { + "type": "integer", + "title": "Additions" + }, + "deletions": { + "type": "integer", + "title": "Deletions" + }, + "files": { + "type": "integer", + "title": "Files" + } + } + }, + "collaborators": { + "type": "array", + "title": "Collaborators", + "visibleOn": [], + "items": { + "type": "object", + "additionalProperties": true, + "properties": { + "github": { + "type": "string" + }, + "roles": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + }, + "ai": { + "type": "object", + "title": "AI usage", + "additionalProperties": true, + "properties": { + "used": { + "type": "boolean", + "title": "AI used" + }, + "level": { + "type": "string", + "title": "Level", + "description": "none | assisted | mostly | full" + }, + "commits_ai": { + "type": "integer", + "visibleOn": [] + }, + "commits_total": { + "type": "integer", + "visibleOn": [] + }, + "body_marker": { + "type": "boolean", + "visibleOn": [] + } + } + } + } + } + }, + "participants": { + "type": "array", + "title": "Participants", + "visibleOn": [ + "read" + ], + "items": { + "type": "object", + "additionalProperties": true, + "properties": { + "github": { + "type": "string", + "title": "GitHub" + }, + "np_user_id": { + "type": [ + "integer", + "null" + ], + "title": "NP user", + "description": "null when the GitHub handle has no mapped nullplatform user" + }, + "roles": { + "type": "array", + "title": "Roles", + "items": { + "type": "string" + } + } + } + } + }, + "ai_usage": { + "type": "object", + "title": "AI usage (aggregate)", + "additionalProperties": true, + "visibleOn": [ + "read" + ], + "properties": { + "prs_ai": { + "type": "integer", + "title": "PRs with AI" + }, + "prs_total": { + "type": "integer", + "title": "PRs total" + }, + "commits_ai": { + "type": "integer", + "title": "Commits with AI" + }, + "commits_total": { + "type": "integer", + "title": "Commits total" + } + } + } + } + } +} diff --git a/deploy-governance/templates/checklist-template-cross-validation.yaml b/deploy-governance/templates/checklist-template-cross-validation.yaml new file mode 100644 index 0000000..7b9a658 --- /dev/null +++ b/deploy-governance/templates/checklist-template-cross-validation.yaml @@ -0,0 +1,101 @@ +# Target template (Gabriel's cross-validation structure, 2026-07-31). +# REQUIRES in prod approval-api: +# - main-approval-api#258 (manual inputs + validations, group aggregation: any) +# - main-approval-api#260 (orchestrator token scope) +items: + - id: change_analysis + type: external + behavior: informational + severity: info + title: "Change analysis" + description: > + Diff between the deployed release and this candidate: commits, PRs, + participants, risk signals + LLM score. Informational — the decision + lives in the cross-validation group below. + external: + kind: deploy-change-analysis + trigger: auto + timeout_seconds: 600 + token_scope: orchestrator + inputs: + application_id: "{{ context.application.id }}" + release_id: "{{ context.release.id }}" + scope_id: "{{ context.scope.id }}" + deployment_id: "{{ context.deployment.id }}" + + - id: cross_validation + type: group + behavior: gate + aggregation: any # ONE of the children passing approves the deploy + title: "Cross validation — one of" + children: + - id: already_deployed + type: external + behavior: gate + severity: info + title: "Already deployed release" + description: > + Passes when this release was deployed to this scope within the + last 10 finalized deploys — fast rollbacks and config re-deploys + skip everything else. + external: + kind: deploy-already-released + trigger: auto + timeout_seconds: 900 + + - id: risk_matrix + type: external + behavior: gate + severity: major + title: "Risk matrix" + description: > + Computed after the change assessment: auto-passes when the light + matrix (change risk x app criticality) allows it. When it does + not, one of the human reviews below unblocks the deploy. + external: + kind: deploy-risk-matrix + trigger: auto + timeout_seconds: 900 + + - id: pair_review + type: manual + behavior: gate + severity: major + title: "Pair review" + description: > + Cross approval by a peer: anyone in the org EXCEPT the requester. + Use when the risk matrix asks for a second pair of eyes. + validations: + - id: four_eyes + rule: + actor.user_id: { $ne: "$approval.requested_by" } + message: "Un par distinto del requester debe aprobar este deploy" + # Validación EXTERNA evaluada AL MOMENTO del submit (no al crear el + # run): el workflow deploy-review-level lee la metadata ya escrita + # por el análisis y solo deja pasar el pair review si la matriz + # pidió nivel "par". Así se "enciende" el item correcto sin + # recalcular conditions: la revalidación ocurre cuando el humano + # actúa, con la metadata ya existente. + - id: review_level + external: + kind: deploy-review-level + timeout_seconds: 120 + inputs: + deployment_id: "{{ context.deployment.id }}" + required_level: "par" + on_timeout: fail + message: "La matriz pidió otro nivel de aprobación para este deploy (ver Change analysis)" + + - id: admin_review + type: manual + behavior: gate + severity: major + title: "Admin review" + description: > + Approval by the platform admin group. Always available and + overrides the matrix for high-risk deploys. + validations: + - id: admin_only + rule: + actor.effective_roles: { $in: ["admin", "ops", "secops"] } + message: "Solo admins/ops pueden usar esta aprobación"