feat: deploy-governance suite — change analysis, backfill, weekly summaries - #22
Open
geisbruch wants to merge 9 commits into
Open
feat: deploy-governance suite — change analysis, backfill, weekly summaries#22geisbruch wants to merge 9 commits into
geisbruch wants to merge 9 commits into
Conversation
…maries Per-deploy change analysis resolving checklist external items (diff, PRs with collaborators and LLM summaries, risk signals + floors, risk x criticality matrix), a self-healing nightly backfill, and weekly functional summaries per app/namespace stored as rolling catalog metadata. Includes catalog-spec setup and the target cross-validation checklist template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
…ops users beyond the first page, leaving their commits unmapped Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
…nalyzer
Participants are stored as {github, roles}; the mapping to platform users
(user/identity metadata) is a lake join in dashboards. Analyses drop ~50
API calls each, and a mapping fixed later applies retroactively to the
whole history at query time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
Per-deploy analyses emit a one-sentence short_summary (heuristic fallback: first sentence); weekly summaries emit up to 3 ultra-short bullets per app and namespace, plus latest_short at the top level. The checklist item resolution message now uses the short form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
Each PR carries ai: {used, level none/assisted/mostly/full, commits_ai,
commits_total, body_marker}, detected from Co-Authored-By trailers and
PR-body markers; deploys get an ai_usage rollup and weekly entries count
ai_prs. The approver-facing markdown tags AI-assisted PRs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
…ot resolve vars at activation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
Each PR records first_commit_at (oldest commit via /pulls/N/commits — the true first line even under squash-merge), opened_at, merged_at and size (additions/deletions/files from the PR detail). Lead times are computed at report time against the PR's first production deploy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V
The spec was an open {type: object} — the dashboard could only dump a JSON
blob. It now carries the full document schema (risk, summaries, categories,
matrix decision, releases, signals, PRs with AI usage, participants) as
specs/deployment-change.spec.json, the single source of truth the setup
script upserts (PATCH in place when the spec exists).
Visibility: visibleOn [read] at the root — renders on the deployment detail
only, never as list columns nor in create/update forms. Raw payloads
(commits, files, long PR descriptions) stay accepted on writes via
additionalProperties but are deliberately undeclared so the schema-driven
UI does not render them; summary_md already narrates them.
The metadata service validates every write against this schema, so it was
calibrated against real stored docs (null-safe from_release / np_user_id /
PR fields, free-string category) and verified with a no-op rewrite of a
live document. Applied to organization=4 on 2026-08-07.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New suite: the deploy-governance system nullplatform dogfoods on its own org.
What's in it
deploy-change-analysis.yaml— resolves checklist external items (kinddeploy-change-analysis) on production deploys: GitHub diff of deployed→candidate release, PRs as rich objects (author, collaborators+roles, LLM one-line summaries), deterministic risk signals with hard floors, and a light risk × criticality matrix. Persists everything todeployment.metadata.change.deploy-change-analysis-manual.yaml— same pipeline behind a plain webhook (smokes + backfill children).deploy-backfill.yaml— nightly cron + manual webhook: finds prod deploys missing metadata and fires the analysis per deploy. Skip-existing → idempotent and convergent under sandbox-pool saturation.deploy-weekly-summary.yaml— weekly cron: aggregates the week's per-deploy summaries into app summaries + namespace roll-ups with ONE LLM call, stored as rolling 53-week catalog metadata (latest_summaryvisible in the UI card, history via API/lake).setup/01-catalog-specs.sh— catalog specs incl. thevisibleOnUI pattern and the named-enum criticality select.templates/checklist-template-cross-validation.yaml— target template: analysis (informational) +cross_validationgroup (aggregation: any) with fastpath / risk-matrix / four-eyes pair review / admin escalation.Security review done before upload
secrets.CHANGE_ANALYSIS_WEBHOOKconfig entry; org id parameterized asvars.NP_ORGANIZATION_ID; scanned for keys/tokens/internal ids/emails.npx np-workflow validate(both graph passes).Why no
__tests__: this suite's I/O happens insidecode-execsandboxes (raw fetch to NP API/lake/GitHub), not through stubbable integration plugins — the README documents this and the smoke path via the manual webhook variant.The README also encodes the production gotchas we hit running the 123-deploy backfill on our own org (sandbox saturation convergence, parallel GitHub reads,
\Nfirst-deploy sentinel, agent prompt-size and tool-detour limits, AJV-validated metadata writes).🤖 Generated with Claude Code
https://claude.ai/code/session_01QMGMC1jJWHdLigWpYg4G5V