Add a release-manager agent that orchestrates the release skills - #1794
Open
jeffhandley wants to merge 16 commits into
Open
Add a release-manager agent that orchestrates the release skills#1794jeffhandley wants to merge 16 commits into
jeffhandley wants to merge 16 commits into
Conversation
Introduces a `release-manager` custom agent that owns the C# SDK release
process end to end, and a `verify-release` skill for the final stage.
The agent routes to the prepare-release, publish-release, and verify-release
skills rather than reimplementing their mechanics, and adds the orchestration
layer around them:
- Five ordered stages (prepare, review-and-merge, publish, release, verify)
with an explicit human gate at each transition
- A progress rail rendered at every gate
- Status reconstruction from repository evidence so a release can resume
across sessions and machines
- Session tracking via the SQL tool for stage timing and gate interactions
- A closing release summary that reports stage timing, total session
wall-clock, and an estimate of active user-interaction time versus wait time
The agent stays on the branch its session started on and delegates the stages
that create commits to a child session on a worktree based on the target
release branch, mirroring how the docs workflow builds each version from its
own worktree while orchestration runs from a single fixed checkout. This
leaves the orchestrator's working tree clean for its long-lived session state
and confines an abandoned preparation to a disposable worktree. Human gates
stay with the orchestrator; the child prepares and reports, and never pushes
or opens a pull request on its own initiative.
The new verify-release skill covers the two workflows that publishing a
GitHub release triggers in parallel: Release, which publishes the NuGet
packages, and Publish Docs, which rebuilds the versioned documentation site.
It confirms both workflow runs, the package listings on NuGet.org, and the
docs site, and it distinguishes propagation lag and concurrency-superseded
docs runs from genuine failures.
Release notes now link to the version-slugged versioning page,
`/v{MAJOR}/versioning.html`, rather than the unslugged URL. The slug follows
the MAJOR of the version being released rather than the branch, so it stays
correct even if the two ever disagree. The unslugged URL tracks the site's
default version and would silently repoint a shipped release's notes once a
later MAJOR ships.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
Adds a Step 0 to the prepare-release and publish-release skills that fetches the upstream remote's branches and tags before anything reads repository state, and teaches the release-manager agent to delegate stage 1 to a fresh worktree based on the upstream ref rather than a possibly-stale local branch. Stale refs do not fail loudly. A missing tag makes a published release look like it never happened, and a stale branch hides merged PRs, so the release assessment comes out confident and wrong. Observed in practice: a checkout missing the `v2.0.0` tag's history reported the tag as being on divergent history and produced 312 ApiCompat errors for missing API surface, which reads as a catastrophic breaking change rather than a fetch that never ran. Step 0 identifies the remote pointing at the canonical repository rather than assuming `origin`, since a fork-based checkout typically has `origin` pointing at the fork. Subsequent steps read from remote-tracking refs. Step 2 of prepare-release now verifies the previous release tag is an ancestor of the target commit and stops if it is not, since a divergent source branch invalidates both the PR range and the ApiCompat baseline. Step 7 warns that a large number of missing-API errors indicates a baseline that does not belong to the branch's history, and that suppressing them is never the answer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
Adds a dedicated release-notes review gate to the prepare-release skill and fixes the two categorization rules that produced corrections during a live release. Showing the finished notes at the push/PR gate does not work. In practice the complete, well-formatted notes were presented and approved, and the content corrections arrived afterward against an already-open PR. Well-formatted notes read as correct and do not invite scrutiny, so the review has to be shaped as a set of decisions rather than a document to skim. New Step 10b presents two compact artifacts and stops for a response after each: a table of every PR with its assigned section and the reason for it, and a roster of who is acknowledged and why. It names the close calls explicitly rather than waiting to be asked, and shows excluded acknowledgements so the user can overrule an omission. Step 12 now confirms this review happened rather than absorbing it. Two rules changed behind that gate: Categorization now tests whether the shipped packages changed, not whether the PR contains code. The previous "code AND docs goes in What's Changed" rule swept up sample-only and behavior-clarifying PRs, which inflates the apparent scope of a release. A PR that adds an entire sample application is still a documentation update, because nothing in `src/` shipped differently. Borderline calls now resolve toward Documentation Updates and get surfaced. Maintainers are no longer acknowledged as issue reporters. Acknowledgements exist to thank the community, and a maintainer filing an issue in their own repository is ordinary project work. They still appear in the reviewers bullet. publish-release carries forward the categorization decisions made during preparation instead of re-deriving them, so a correction the user already made is not silently reverted when the notes are refreshed for late-arriving PRs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
Adds a mandatory suppression audit when PackageValidationBaselineVersion changes, and documents the diagnostic misreading that turned a one-method additive release into an apparent 312-break emergency. Suppression entries are scoped to the baseline they were generated against: they record that a difference from *that* baseline is intentional. Moving the baseline changes the set of differences, so entries written for the old baseline can describe nothing at all. ApiCompat reports those orphans, and the build fails. The failure is easy to misread. `Unnecessary suppressions found` is itself a hard failure, and the CP0001/CP0002/CP0005 lines printed beneath it are the tool's listing of unused suppression entries -- not live API breaks -- even though they are formatted identically and appear under the same banner. Moving the baseline from 1.4.1 to 2.0.0 made 312 Core Tasks suppressions stale, and that listing read as a catastrophic regression on a release whose only real change was one additive method. Two independent checks distinguish the two cases, and the guidance now requires them before a release is classified as breaking: regenerate the suppression file with ApiCompatGenerateSuppressionFile, where empty output proves every tracked entry is stale, and cross-check the direct API diff. Both were unambiguous here. The audit runs per shipping project against the final candidate version and baseline, writes to a throwaway file so the tracked one survives the investigation, and requires a plain CI-equivalent pack to pass afterward with no generation flags. Reverting the baseline is documented as an equally valid resolution, to be chosen deliberately rather than by whichever option silences the error first. Adds an explicit prohibition on tuning the validation to pass -- changing the baseline to clear a red build, ApiCompatPermitUnnecessarySuppressions, NoWarn for CP diagnostics, or disabling package validation. These hide the signal that the baseline and the suppressions have drifted apart, which is precisely what needs to be known. Corrects the suppression-file wiring guidance: a project-directory CompatibilitySuppressions.xml is auto-discovered and needs no wiring, CompatibilitySuppressionFilePath is the supported property for an explicit path, and ApiCompatSuppressionFile is an item rather than a property, so setting it via /p: does nothing. A retained empty file must be valid XML and preserve the repository's BOM and final-newline conventions. The Step 12 summary now reports per shipping package: baseline version, generated suppression count, retained versus removed stale entries, and the plain-pack result. "ApiCompat passed" is not reportable without them, since it says nothing about what was validated against or whether stale suppressions shaped the outcome. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
Opening the release PR currently ends stage 1 with a URL and an invitation to review. That leaves the user to discover CI failures themselves, which is backwards: the agent is holding the context needed to interpret them, and it just caused a failing Pack/APICompat job to be spotted by the user rather than reported. Creating the PR now starts a watch that runs until every check reaches a terminal state, and restarts automatically on every subsequent push to the release branch. A restart targets the new head SHA -- checks that passed on an earlier commit are stale and must not stand in for the ones now at the head of the branch. Queued and in-progress checks are not results, and a workflow that never started is not the same as one that passed. Monitoring is read-only and needs no permission. Acting on what it finds still goes through the existing gates: a fix is diagnosed, proposed, and pushed only on explicit approval, and committed in the child session rather than the orchestrator. On failure the agent retrieves the logs itself instead of asking for a paste, and classifies before proposing anything, because the two classes call for opposite responses. Product and API validation failures are real and must be diagnosed; rerunning a deterministic failure spends a full CI cycle to arrive at the same red. Infrastructure and tooling failures justify a single rerun with the reason stated. Flakiness requires evidence rather than being the convenient explanation for a log left unread. ApiCompat failures route through the suppression-audit interpretation rules first, since a stale baseline manufactures large and convincing phantom breaks. Reporting is a per-check table plus one verdict of green, running, or blocked, where blocked covers any non-green terminal state including cancelled and timed out. Stage 2 stays blocked until the checks are green or the user explicitly decides otherwise, and that decision is recorded. The handoff leads with CI status rather than only inviting review, since without it the user cannot tell whether the invitation is even actionable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
Stage 4 ended by asking the user to come back and report that they had published. That is the wrong moment to be uninformed: publishing is when the release becomes irreversible and when the Release and Publish Docs workflows both start, so verification that begins whenever the user next speaks arrives after the interesting part. After creating the draft, the agent now polls it until `isDraft` becomes false, at a modest interval since the gate is human-paced and may span hours or a session boundary. On detection it records the stage 4 end time from `publishedAt` rather than from when the poll noticed -- polling latency is the agent's, not the user's, and should not inflate the stage duration in the closing summary -- confirms the tag actually created and the prerelease flag, both of which were the user's to set and cannot be inferred, and starts stage 5 on its own. That transition is announced rather than requested: verification is read-only, and the irreversible act has already happened. The watch also distinguishes outcomes that are easy to conflate. A draft whose body changed but is still a draft means the user is reviewing, possibly removing the AI disclosure, and calls for no action at all. A draft that disappears may have been published under a different tag rather than abandoned. A published tag that differs from the prepared version stops the process, because verifying the wrong version is worse than not verifying. And if the user reports publishing while the API still shows a draft, the API is believed -- an unsaved draft is indistinguishable from success in the browser. Renames ci-monitoring.md to monitoring.md, since it now covers both watches. They share a principle worth stating once: monitoring is automatic and read-only, so watching never needs permission, while acting on what it finds always does. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
The v2.1.0 release exposed four ways the timing data went wrong. Every one of them produced a number that looked reasonable, which is worse than an obviously missing one. Interaction rows were left with NULL answered_at, or were recorded as zero-duration when the reply timestamp was not available. Close the open interaction before acting on the reply, never equate answered_at with prompted_at, and report the measured/unmeasured split so the total reads as the floor it is. Waiting was inferred by subtracting interaction time from stage wall-clock, which counts diagnosis and rework as waiting. Log unattended waits and workflow runs as their own records, and surface whatever the two do not account for as unaccounted rather than folding it into either. Stage ends were taken from when the agent noticed an event rather than when it happened, inflating a stage by the polling interval. Take them from mergedAt, publishedAt, and workflow timestamps. A single row per stage hid the shape of the time: stage 2 read as "2h 22m" with no way to tell CI from review from remediation. Stages now carry an attempt, and rework opens a new one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
The stage timing table reported elapsed time alone, so the one number a reader actually wants -- how much of the release cost them their attention -- was stranded in the narrative below it as a single session-wide figure. A stage that took 2h 22m of wall-clock but ~11m of the user's attention reads very differently from one that took 2h 22m of theirs. Splitting the two per stage makes that visible at a glance, and makes it obvious which stages are worth automating further. The roll-up depends on every interaction being attributed to a stage when it is recorded, so that requirement moves next to the schema rather than being implied by the summary. Stages keep their distinction between "~0m", meaning decisions that took no material time, and an em dash, meaning the timestamps were never captured; the total carries a "minimum" qualifier whenever either gap exists. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
The interaction total hedged twice, rendering as "~27m minimum". The tilde already says the figure is estimated, so the extra word only cost the table its scannability -- and a timing table earns its place by being readable at a glance. Timestamp limitations still get reported, just in the prose below the table where there is room to say which stages were affected and why. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
The two timing references disagreed about the same cell. summary-template said a stage with recorded decisions but no material timed interaction shows ~0m, while session-tracking said a stage with interactions but no measurable intervals shows an em dash. Those phrases read as the same condition and gave opposite answers, so the distinction they exist to protect -- zero time versus absent data -- was the first thing to collapse under it. Both now key off whether a usable reply timestamp exists: an em dash when interactions happened but none can be measured, ~0m only when the stage had no interactions or its measured intervals round to zero. That matches what the roll-up query actually returns, which is NULL rather than 0 for the unmeasured case. Also scoped the measured/unmeasured reporting to the narrative. It predated the rule that the table carries nothing but the tilde, and still argued that "~27m" alone overstates precision -- which now reads as an argument for hedging the cell that rule just cleaned up. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
Step 9 displayed a merge commit SHA for review and then created the release
with --target {merge-commit-branch}. A draft is not a snapshot: GitHub stores
the target and creates the tag only at publish, which in the v2.1.0 release was
hours later. Anything landing on that branch in between would have silently
moved the tag to a commit nobody reviewed, while the release notes went on
describing the commit that was.
Targeting the reviewed SHA closes that window at no cost, since the tag still
is not created until publish and the draft stays fully editable. Pulling in a
later commit is now an explicit act -- repoint with gh release edit and
regenerate the notes -- rather than something that happens to you.
Two other corrections found alongside it.
Stage 1's boundary was defined twice and disagreed: session-tracking ended it
at PR creation, monitoring opened by saying that is explicitly not the end of
it. Both stages then claimed the CI watch. Stage 1 now ends when the PR opens
and stage 2 owns everything after, so preparation does not absorb review and
remediation time.
The baseline rule was stated correctly in the reference and then restated in
the step that executes it as a literal from a long-past release, hard-coding
the 2.0.0-preview series to 1.3.0. Refreshing those literals only restarts the
rot, so the step now derives the value from what is actually published and
shows its work. That also resolves a quieter conflict: the edge case insisted
existing suppressions be preserved, which is exactly wrong when the baseline
moved and the audit proves them stale.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
A worktree left behind by an interrupted stage 1 says only that work started. The agent treated its existence as progress, so it could resume onto a branch whose validation had never passed and whose release notes nobody had reviewed. Give it an evidence ladder: read the commit, re-run everything that leaves no trace in the repository, and put recovered decisions back through the user before building on them. Record the child's session, worktree, and branch when it is dispatched, so a later session can find the work rather than infer it. The suppression guidance had the same shape of error. It described a compatibility suppression file as append-only, which is what let 312 obsolete Core entries read as a mass breaking change. Name all three outcomes - added, retained, cleared - and say that preservation holds only while the baseline does. Stop offering a tracked file as a template for entry shape; it is legitimately empty after an audit. Replace the hard-coded three-package enumerations and baseline literals with instructions to enumerate what actually ships. The set grows, and the next package to join validation would have gone unreported. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
PackageValidationBaselineVersion deliberately stays at the last MAJOR release while VersionPrefix advances through the series, so 2.0.0 sitting beside a published 2.1.0 is the rule working. The reference stated the rule but never said what it looks like in practice, and a second independent reader has now concluded the value was stale and worth bumping. Name the trap and the cost of "fixing" it: bumping mid-series re-baselines the released API against itself, discarding the guarantee, and trips the baseline-transition audit that produced the 312-entry incident. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
…swer Previous-release lookup contradicted itself inside a single sentence, saying "most recent published globally" and "highest semver" as though they were the same rule. They diverge the moment a servicing patch ships after a newer minor: v2.0.1 published after v2.1.0 is the most recent by date but is not on main at all, so date ordering picks a tag off the branch, producing a bogus PR range and an ApiCompat run that reports the entire API surface as removed. Settle on highest semver among ancestors of the target, and align the four call sites. Workflow-run correlation matched on event type plus a timestamp at or after publishedAt, which any concurrently published release also satisfies - a green run for the wrong tag. Release-event runs carry the tag in headBranch, so match on that exactly and cross-check headSha against the reviewed target. Stage 3 corrective commits were told to land on the release branch, which is merged and gone by then. They belong on the protected base branch behind their own PR. More importantly, the draft is now pinned to an approved SHA, so a fix merged afterward is not in the tag unless the draft is re-targeted - otherwise the notes describe a fixed state the tag does not contain. Wait and interaction intervals overlapped by construction, since gates get answered mid-CI-watch. Summing both against wall-clock made the parts exceed the whole: a 2h watch containing an 11m gate yielded -11m unaccounted. Keep the two disjoint while recording by closing the wait around each interaction, and refuse to publish a negative or clamped remainder. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
The rule read "do not treat 'here are the finished notes' as a review", quoting a phrase that never appears as literal output and so scanned as a stray fragment. State the behavior directly; the guidance is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
…ople The Step 10b tables and the CI handoff templates were filled in with actual PR numbers, commit SHAs, and contributor handles from the v2.1.0 release. A named maintainer appearing as the worked example of someone to omit from acknowledgements is the worst of these, but all of them share a defect: they read as data rather than as shape, and they go stale the moment those PRs are no longer the ones in flight. Use placeholders and say what each column turns on, so the format is legible without borrowing a real person to illustrate it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b8535f33-08ac-44ef-84d1-aab96770ae3a
tarekgh
reviewed
Aug 5, 2026
| never checks out or mutates a release branch. Work that creates commits happens in a **child session | ||
| on its own worktree**, based on the target release branch. | ||
|
|
||
| This mirrors how [`docs.yml`](../../workflows/docs.yml) already works: the orchestration scripts run |
Contributor
There was a problem hiding this comment.
Broken relative link. This file lives three levels under .github/ (agents/release-manager/references/), so ../../workflows/docs.yml resolves to .github/agents/workflows/docs.yml, which does not exist. It needs one more level: ../../../workflows/docs.yml. Note the ../../../skills/publish-release/SKILL.md link later in this same file already uses the correct depth.
| than reading the log. | ||
|
|
||
| 3. **For ApiCompat and package validation failures specifically**, apply the interpretation rules in | ||
| [apicompat-apidiff.md](../../skills/prepare-release/references/apicompat-apidiff.md) before |
Contributor
There was a problem hiding this comment.
Broken relative link. From agents/release-manager/references/ this ../../skills/... path resolves to .github/agents/skills/prepare-release/references/apicompat-apidiff.md, which does not exist. It should be ../../../skills/prepare-release/references/apicompat-apidiff.md (three levels up to reach .github/). delegation.md in this same directory already uses the correct ../../../skills/... depth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releasing this SDK means driving
prepare-releaseandpublish-releasein the right order, knowing which parts are irreversible, and remembering a lot of hard-won detail about what "green" actually means. That knowledge lived in people's heads, so each release rediscovered the same traps.This adds a
release-manageragent that owns the whole process end to end, plus averify-releaseskill that closes the loop after publishing.Notes
This is documentation/infrastructure only. There are no source or test changes, so nothing here affects the shipped packages.
All of this was produced by capturing experiences from the v1.4.1, v2.0.0, and v2.1.0 releases. The agent was implemented ahead of the v2.1.0 release, and it was used for that release, addressing hiccups and opportunities along the way. Thereafter, a few rounds of agent-assisted and local, human self-review were completed before creating this PR.
Approach
The agent is an orchestrator across five stages: prepare, review and merge, publish, release, verify. It holds the gates and the timing; the skills do the work. Stage 1 is delegated to a child session on a fresh worktree, because preparation writes to a release branch and the orchestrator should not be switching branches mid-release. Stage 4 is a deliberate human action in the GitHub UI that the agent will not perform on the user's behalf.
The agent file carries the always-loaded operating rules. Four references hold the detail:
delegation.md,monitoring.md,session-tracking.md, andsummary-template.md. Session state lives in SQLite so the closing summary reports measured stage timings and interaction time rather than recollection.Why the release skills changed too
This was test driven by actually shipping v2.1.0 with it. Every skill change here traces to something that went wrong or read ambiguously during that release:
Unnecessary suppressions foundis itself the failure, and the CP lines beneath it list unused entries rather than live breaks. There is now an audit procedure, and the guidance says a suppression file can legitimately end up empty.gh release create --targetresolves at tag creation, which happens at publish, potentially hours later. A branch target silently re-resolves to whatever landed meanwhile, cutting the tag at an unreviewed commit while the notes describe a different one. Drafts do not create the tag, so the draft is now pinned to the reviewed SHA and re-targeted explicitly if late commits are taken.headBranch, so that is now the key.Note
This pull request was authored with GitHub Copilot.