Conversation
Confirm fullsend v0.37.0 toolchain, stock claude-runtime image binaries (all present -> TC-5806 no-op), and root-level harness resolution of the in-place sdlc-workflow plugin. Note v0.37.0 constraints for TC-5807: --fullsend-dir must be the repo root for relative children to resolve, and verify-pr is a valid harness role but not a valid config.yaml role (enum: fullsend/triage/coder/ review/fix/retro/prioritize/e2e). Implements TC-5805 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author harness/verify-pr.yaml at the repo root so its relative children resolve against the repo root and plugins/sdlc-workflow is delivered in place as a whole plugin. Declares every field explicitly with no base composition; omits security: (Go defaults supply it) and any forge block (GitHub is tier-1, Vertex-only provider). Split-trust env: Jira/GitHub tokens on the runner only, read-only context (JIRA_ISSUE_ID, JIRA_BASE_URL) in the sandbox. Also adds the sandbox agent prompt (agents/verify-pr.md) and the Vertex env file (env/gcp-vertex.env). Referenced children (providers, policy, profile, schema, pre/post scripts) are authored by later epic tasks; this task validates the harness syntactically only. Implements TC-5807 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vertex is the sole in-sandbox provider (tier 4). Jira and GitHub are tier-1 (prefetched host-side) and need no in-sandbox provider, so only the Vertex pair is declared locally for the standalone harness. Copied verbatim from the stock agents Vertex pair: provider type fullsend-vertex-ai matches profile id fullsend-vertex-ai (endpoint *.googleapis.com:443), which the root harness/verify-pr.yaml references via providers:/openshell.profiles:. Implements TC-5808 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add plugins/sdlc-workflow/policies/verify-pr.yaml for the verify-pr fullsend harness (referenced by harness/verify-pr.yaml). Filesystem is read-only (include_workdir: false); egress is reduced to Anthropic + Vertex AI (*.googleapis.com) inference and telemetry only. No *.atlassian.net (Jira) and no api.github.com (GitHub) egress: those are tier-1 with runner-only tokens under the split-trust I/O model (prefetch/post-script run host-side). curl and gh are excluded from the binary allowlist to block raw HTTP with injected tokens. Live fullsend egress verification is deferred to TC-5810/TC-5811, which author the harness's remaining siblings (pre/post scripts, result schema). Implements TC-5809 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The verify-pr fullsend policy granted Vertex AI (*.googleapis.com) egress but omitted the pi runtime from the binary allowlist. Under fullsend's dual host-AND-binary match rule, the pi runtime — which brokers Vertex inference — would be blocked. Add **/pi alongside **/claude and **/node to match the composed fullsend-vertex-ai profile and every upstream vertex policy. Implements TC-5880 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on_loop
Add the split-trust I/O contract for the verify-pr fullsend harness: the
pre_script prefetches everything on the trusted runner (where the Jira and
GitHub tokens live) and the sandbox reads only JSON — no api.github.com or
atlassian egress.
- schemas/verify-pr-input.schema.json — tracker-agnostic pre-script input,
extended with a `github` tier-1 read bundle (diff, stat, reviews,
review_comments, issue_comments, commits, headRefName, commit_sha).
- schemas/verify-pr-result.schema.json — agent result contract (ported).
- scripts/pre-verify-pr.sh — validates env, fetches the Jira issue, resolves
the linked PR, prefetches every GitHub read the skill performs, and writes
verify-pr-input.json for host_files to mount.
- scripts/pre_verify_pr.py — PR-URL extraction + input transform, extended
with build_github_bundle and a --github-dir transform mode.
- scripts/{validate-output-schema.sh,strip_extra_properties.py} — kept per the
Step 5 native-validator decision (below); wired via validation_loop.script.
- scripts/test_pre_verify_pr.py — unit tests incl. the github bundle.
- harness/verify-pr.yaml — set validation_loop.script.
Deviations from the literal task steps, dictated by the verified fullsend
v0.37.0 contract (same class as TC-5805):
- Skip signal (step 4) uses the pre-script output protocol v1 — line-based
`skipped=true` / `reason=...` appended to $FULLSEND_PRESCRIPT_OUTPUT (guarded,
exit 0) — not a JSON `{"skipped":true}` document. That env var is the
skip-signal file, not the input path.
- GitHub bundle is embedded in verify-pr-input.json (single file) rather than
separate in-sandbox file paths, keeping the harness change scoped to
validation_loop; input JSON stays at /tmp/fullsend-pre-output (host_files src),
with PRE_DIR overridable for host-side testing.
- No host-side PR-head checkout (step 3): the prefetched PR diff is sufficient
for verification, so the sandbox needs no writable checkout.
- Native validator decision (step 5): additionalProperties:false rejects benign
agent extras, so validate-output-schema.sh + strip_extra_properties.py are
KEPT and validation_loop.script is set (fullsend also requires a script for
validation_loop — schema alone is insufficient).
Validated host-side: both schemas are valid JSON; 15 unit tests pass; the
validator strips benign extras and passes, and fails a bad enum (exit 1); the
skip path exits 0 with the correct signal; the happy path (stub gh + jira)
produces a verify-pr-input.json that validates against the input schema with
the full github bundle embedded.
Implements TC-5810
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assisted-by: Claude Code
gh pr diff has no --stat flag, so the prefetch aborted under set -euo pipefail before writing verify-pr-input.json. Derive the per-file stat from the already-fetched patch with git apply --stat, guarding the empty-diff case so the script still exits 0. Add regression tests exercising the real stat command and guarding against reintroducing the unsupported gh flag. Implements TC-5884 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
The GitHub tier-1 prefetch fetched PR reviews, review comments, and issue comments without --paginate, so GitHub REST returned only the first ~30 items and the bundle silently truncated on active PRs — degrading the very review-comment analysis it feeds the verify-pr sub-agents. Add --paginate --slurp and merge pages with `jq 'add'` into the flat arrays that build_github_bundle expects. (--slurp is incompatible with gh's built-in --jq, so a standalone jq performs the merge.) Also make the sibling git-apply-stat test deterministic by running it from a non-repo cwd: git apply --stat is CWD-sensitive and reports "0 files changed" when launched from inside a repo subdirectory. Implements TC-5885 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
transform_to_input set task.description via fields.get("description", {}),
whose default applies only to an ABSENT key. When Jira returns an explicit
null (an issue with no description), .get returns None, so task.description
became null — violating verify-pr-input.schema.json (which requires an
object) and producing a schema-invalid verify-pr-input.json. Use
`fields.get("description") or {}` to also coerce null to {}, matching the
existing `(fields.get("status") or {})` idiom in the same transform.
Add regression tests: a direct coercion assertion and a full-input
(task + github bundle) validation against verify-pr-input.schema.json that
fails on the pre-fix null description.
Implements TC-5886
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The github.com PR-URL match at pre-verify-pr.sh:100 was start- but not end-anchored, so a malformed value like `.../pull/42abc` or `.../pull/42/invalid` matched and BASH_REMATCH truncated the pull number to 42 — making the pre-script prefetch and embed a different PR than the Jira custom field identifies. Append `/?$` so the full URL must match: a trailing non-numeric character or extra path segment is now rejected with the existing "not a github.com pull request URL" error, while a canonical URL with or without a trailing slash still parses to owner/repo and number. Add regression tests that run the script's actual regex (extracted from source) through bash's [[ =~ ]], asserting well-formed URLs parse to the correct owner/repo and number and malformed ones are rejected rather than truncated. Implements TC-5891 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gh's `pr view` commits connection is bounded, so `.commits[-1].oid` returns the last commit of a truncated page rather than the PR head on large PRs — a silently-wrong commit_sha that still satisfies the result schema's hex pattern. Read the head ref tip OID directly (`--json headRefOid`), which is correct regardless of commit count. Also document that the bundled commits list uses the same bounded connection (best-effort context; head SHA is authoritative). Adds a behavioral regression test that runs the shipped COMMIT_SHA command against a gh stub whose headRefOid and commits[-1].oid disagree, plus a source guard, both of which fail on the old logic and pass on the new. Implements TC-5924 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rivations Two field descriptions in the `github` read bundle of verify-pr-input.schema.json cited commands that no longer produce their values (doc-string drift only; the field values are correct): - `commit_sha` cited `.commits[-1].oid`, but pre-verify-pr.sh derives it from `gh pr view --json headRefOid` (TC-5924 — the pr-view commits connection is bounded, so `.commits[-1]` can be a truncated-page tip). - `stat` cited `gh pr diff --stat`, an unsupported flag; it is derived from `git apply --stat` on the already-fetched pr.diff (TC-5884). Documentation-only; the schema remains valid JSON. Implements TC-5929 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port the verify-pr write executor to run host-side after the sandbox is
destroyed. Jira comments and the verification report are posted through the
native `fullsend issues post-comment` sticky-comment CLI (idempotent via a
stable marker), while the irreducible custom Jira writes with no native
primitive yet — sub-tasks, links, and root-cause tasks — call jira-client.py
directly. GitHub writes stay host-side on `gh`.
- post-verify-pr.sh: locate the latest agent-result.json, validate, delegate
- execute-actions.py: resolve {{ref.key}}/{{ref.url}} placeholders, route
post_comment/post_report Jira side through the native CLI, keep gh for
GitHub, keep jira-client.py for sub-tasks/links/root-cause
- jira-client.py: create_issue accepts a pre-rendered ADF description and an
optional parent key (reconciled additively; existing callers unaffected)
- test_execute_actions.py: ref-resolution tests plus native-CLI argv/env,
stdin body, and post_comment/post_report routing tests
Implements TC-5811
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ira CLI execute_post_comment read action["body_md"], but the result schema requires body_adf (an ADF object) and strip_extra_properties.py removes any extra key, so every schema-valid post_comment raised KeyError before posting. Add adf_to_markdown to render the ADF body back to the markdown the native fullsend CLI consumes on stdin, and update the masking test to feed a schema-valid body_adf action. TC-5930 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
post-verify-pr.sh only located agent-result.json, but its sibling validate-output-schema.sh accepts result.json as a fallback. A result that passed validation via the fallback name was then rejected by the write path. Prefer agent-result.json per iteration output dir, and fall back to result.json when absent — matching the validator's precedence. Implements TC-5931 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
post-verify-pr.sh selected the "most recent" iteration by keeping the last match in shell glob order, which is lexicographic. Once there are >= 10 iterations, iteration-9 sorts after iteration-20 and a stale iteration is chosen, so the write path would execute the wrong (older) result. Iterate the iteration-*/output directories in ascending numeric order via `sort -V` and keep the highest-numbered one that has a result file. Preserves the `set -euo pipefail` behavior and the result.json fallback (TC-5931); a `[[ -d ]]` guard skips the literal glob pattern when no iteration dir exists. TC-5932 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ADF block renderer had no taskList/taskItem case, so a schema-valid post_comment task list fell through to the unknown-block fallback and was flattened to plain paragraph text, losing the checklist markers. Add _render_adf_task_list rendering DONE/TODO items as markdown checkboxes, handling both paragraph-wrapped and inline taskItem content. The comment route calls adf_to_markdown directly (not sanitize_adf), so the renderer is the correct fix site. Implements TC-5936 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… retry execute_post_report posted the verification report to GitHub via a plain gh pr comment (no sticky marker) before the idempotent Jira post, so a retry after a failed Jira post double-posted the GitHub report comment. Embed a commit-scoped marker in the GitHub report body and, before posting, list the PR comments and PATCH-update an existing same-commit report comment instead of creating a duplicate. Dedup is scoped to the commit SHA so a later commit still gets a fresh comment, preserving the per-run verification history. The Jira side is unchanged and receives the clean body. Updated the module docstring so its idempotency claim matches actual behaviour. Implements TC-5937 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_find_report_comment_id listed a PR's issue comments with `gh api ... --paginate` and parsed the result with a single json.loads. Without `--slurp`, `gh --paginate` concatenates one JSON array per page (`[...][...]`), which is not valid combined JSON once the PR has more than one page of comments (>30). json.loads then raised JSONDecodeError, which the code swallowed by returning None; execute_post_report read that as "no existing report comment" and created a duplicate GitHub report comment instead of PATCH-updating the existing one, silently defeating the retry idempotency TC-5937 introduced. Add `--slurp` so gh emits a single array-of-pages and flatten the pages into one comment list, so a same-commit report comment is found even when it lands on a later page. Stop swallowing JSONDecodeError: with `--slurp` a parse error is a real failure and now surfaces via sys.exit(1) rather than being misread as "no existing comment". Added a multi-page test (report comment on a non-first page → PATCH-update, not duplicate) and a parse-failure test (exits instead of returning None). Implements TC-5948 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…teral text _render_adf_inline emitted a text node's literal value verbatim into the markdown handed to the native `fullsend issues post-comment` CLI. Because that CLI re-parses the markdown, literal markdown-active sequences inside ADF text (`*`, `_`, `[`, `]`, backticks, and backslash) were reinterpreted as formatting — e.g. a literal `*note*` rendered as emphasized text and `[x](y)` as a link. Add _escape_markdown, which backslash-escapes those characters (backslash first so its escapes are not re-escaped), and apply it to a text node's plain value before the mark wrapping the renderer intentionally adds, so the mark syntax is not double-escaped. Inline code is exempt: its content is literal to the CLI and escaping would corrupt inline-code semantics. Link hrefs are also left untouched. Added tests: literal `* _ [ ]` and backtick are escaped; strong/link marks, inline-code content, and a link href with an underscore are unaffected. Implements TC-5949 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_render_adf_inline previously handled only text and hardBreak nodes; every other inline node fell into the else branch and recursed into a nonexistent content array, so mention/emoji/inlineCard/date/status inline nodes (whose value lives in attrs, not content) rendered as empty strings and were silently dropped -- contradicting the module's own _INLINE_NODE_TYPES set. Render each from its attrs: mention/emoji/status from attrs.text (escaped as literal text; emoji falls back to attrs.shortName), inlineCard from attrs.url (bare, unescaped), and date from attrs.timestamp via a new _render_adf_date helper (epoch millis to UTC YYYY-MM-DD). The final else is kept for genuinely unknown container-like inline nodes. Works in both paragraph and taskItem inline contexts. Implements TC-5950 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…result schema
The post_comment action schema accepted any non-empty string for "issue", but the executor post_jira_comment_native requires a hyphenated Jira key and sys.exit(1)s otherwise — a schema-valid {"issue":"12345"} (numeric ID, URL, or other non-key identifier) passed validation and then failed at execution, aborting the whole post_script run.
Constrain the schema "issue" field to the Jira-key pattern ^[A-Z]+-[0-9]+$ (matching jira_issue_id/parent in the same schema) so non-key identifiers are rejected at the producer boundary. The executor rpartition fail-fast guard is kept as defence in depth. Add schema-validation tests asserting the pattern accepts a valid key and rejects non-key issues (numeric ID, URL, lowercase, missing hyphen/number/project).
Implements TC-5953
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 9 'Post to GitHub PR' claimed every verification run creates a new PR comment and never overwrites previous reports, and showed a plain `gh pr comment` create. That contradicted the shipped executor behavior (TC-5937): execute_post_report embeds a commit-scoped marker and, via _find_report_comment_id, PATCH-updates an existing same-commit report comment instead of duplicating it — a new comment is created only for a later commit. Rewrite Step 9 to match the code: describe the commit-scoped marker, the find-then-PATCH-or-create path (gh api --paginate --slurp + PATCH, else gh pr comment), and reframe the history claim as per-commit (a later commit gets a new comment) rather than per-run. Keep the (commit <short-sha>) header convention. Add an eval assertion (evals/verify-pr/evals.json case 3) covering the commit-scoped report header / per-commit history. Documentation-only alignment; executor behavior is unchanged. Implements TC-5954 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…schema
TC-5953 tightened the post_comment 'issue' field to the Jira-key pattern
'^[A-Z]+-[0-9]+$', but execute_post_comment resolves the field through
resolve_refs, which supports the {{<ref>.key}} placeholder so a comment can
target an issue created by an earlier action. Since fullsend's validation_loop
validates the result schema before post_script runs, a placeholder issue was
rejected before the reference could be resolved.
Widen the pattern to '^([A-Z]+-[0-9]+|\\{\\{[a-z0-9-]+\\.key\\}\\})$' so it
accepts either a literal Jira key or a {{<ref>.key}} placeholder, scoped to
.key (not .url) since the field must resolve to a key. Bare numeric IDs, URLs,
and other non-key/non-ref identifiers stay rejected (TC-5953's intent).
Add a schema-validation test that a {{<ref>.key}} post_comment action passes
jsonschema.validate, and extend the negative test with .url placeholders,
uppercase refs, and unanchored/brace-less forms.
Implements TC-5959
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
datetime.fromtimestamp can raise OverflowError/OSError (or ValueError) for an out-of-range epoch-ms value. The call sat outside the guarded try, so a single malformed ADF date node in a post_comment/post_report body aborted the entire execute-actions.py post_script and posted nothing. Move the fromtimestamp call inside the try and catch OverflowError/OSError alongside the int() parse errors, falling back to the literal timestamp string so the node degrades gracefully. Implements TC-5965 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
create_issue declared issue_type: str = "" (reconciled under TC-5811 for the
new description_adf/parent params), so an omitted value serialized to
{"name": ""} and surfaced as an opaque Jira 400 instead of failing fast at
the call site. Restore the prior fail-fast contract: reject an empty (or
whitespace-only) issue_type with a clear stderr message and sys.exit(1)
before any HTTP request, matching the module's existing error style.
Added test_create_issue_fails_fast_on_empty_issue_type (asserts SystemExit
and no request issued); jira-client suite 19 -> 20 passing.
Implements TC-5966
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
post_comment and post_report both posted via post_jira_comment_native with the single shared STICKY_COMMENT_MARKER, which the native fullsend CLI treats as the sticky-comment identity. Two comments targeting the same Jira issue in one run would share that marker, so the second post would overwrite the first and silently lose a comment. Thread a marker parameter through post_jira_comment_native (default preserves the report path's historical marker) and give post_comment a distinct POST_COMMENT_STICKY_MARKER. Each path keeps its own stable marker, so per-path re-run idempotency is preserved while cross-purpose clobbering is prevented. GitHub report dedup (GITHUB_REPORT_MARKER_PREFIX) is untouched. Added test_post_comment_and_report_use_distinct_sticky_markers; execute_actions suite 26 -> 27 passing. Implements TC-5967 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
execute_post_report embedded the raw commit_sha in the GitHub dedup marker and _find_report_comment_id matched it by substring. The result schema permits commit_sha to be 7-40 hex chars, so a run recording a full SHA and a retry recording an abbreviation (or vice-versa) for the same commit produced different marker strings; the substring lookup missed and the retry posted a duplicate report comment, defeating the TC-5937 idempotency guarantee. Normalize the SHA to the schema-minimum 7 chars in the marker (and thus the lookup, which reuses the same marker). Git abbreviations are always prefixes of the full SHA, so first-7 is the only fixed length that unifies a full SHA with any valid abbreviation of the same commit; a longer length (e.g. 12) would leave a 7-char abbreviation unchanged and still mismatch a full SHA. Added test_execute_post_report_dedup_marker_invariant_to_sha_length (full-SHA create then abbreviated-SHA retry -> single PATCH-updated comment); execute_actions suite 27 -> 28 passing. Implements TC-5968 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
execute_post_report calls resolve_refs on report_md, which raises an
uncaught KeyError for any {{ref.key}} placeholder whose entity has not yet
been registered. The registry is populated by create_subtask /
create_root_cause_task as main()'s sequential loop runs, so a post_report
ordered before an action it references would abort the whole run with a bare
KeyError. verify-pr emits post_report last today, but that ordering was
undocumented and unenforced.
Defer every post_report until after the actions loop in main() so the
registry is fully populated before any report_md is resolved. post_report is
still a recognized action; only its execution moves to the end. Observed
behavior is unchanged for the always-last emission order.
Added test_post_report_resolves_ref_created_by_later_action, which drives
main() with a temp result JSON that orders post_report before the
create_subtask its report_md interpolates and asserts the resolved key
reaches the GitHub report body; execute_actions suite 28 -> 29 passing.
Implements TC-5969
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_render_adf_block handled paragraph/heading/rule/codeBlock/bullet+ordered list/taskList; every other block hit the unknown-block fallback that recurses into content and flattens structure. So table cells were concatenated with no grid, blockquote/panel lost their framing, and media/mediaSingle/mediaGroup (which carry no text content) were dropped entirely. A schema-valid post_comment.body_adf can carry any of these, so a comment silently lost content — the same silent-degradation class already closed for taskList (TC-5936) and inline leaf nodes (TC-5950). Add explicit renderers: blockquote/panel -> "> "-prefixed lines (panelType surfaced as a bold label), table -> a GitHub-flavored markdown table (first tableRow is the header, pipes in cells escaped), and media/mediaSingle/ mediaGroup -> an image link or a non-empty [alt] placeholder, never dropped. The recursing fallback is kept for genuinely unknown container nodes, and the adf_to_markdown docstring's node-set contract now lists the covered nodes. Added rendering tests for table, blockquote+panel, and media, plus one asserting an unknown block still degrades via the fallback; execute_actions suite 29 -> 33 passing. Implements TC-5970 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…us alias) Under fullsend CLI v0.43.0 the `opus` alias resolves to claude-opus-4-6, which the Red Hat Vertex deployment has not enabled (run 34951983374 failed: "model claude-opus-4-6 is not available on your vertex deployment"). Pin to the fleet's target generation, opus 4.8. Refs TC-6186. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Carries the harness model pin (claude-opus-4-8) into the URL-pinned base the CI fetch resolves. Refs TC-6186. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sandbox no longer receives JIRA_ISSUE_ID (dropped from env.sandbox in harness/verify-pr.yaml by TC-6190), but the agent prompt still read $JIRA_ISSUE_ID and passed a hardcoded TC-4715 arg, so the sandbox started with an empty issue ID. Derive the Jira key from the pre-fetched verify-pr-input.json `task_id` (a required schema field the pre_script resolves from the triggering PR URL) and fail fast if it is missing. Update fullsend.md accordingly: the sandbox gets the key via task_id, the local-run entry point is FULLSEND_WORK_ITEM_URL (not JIRA_ISSUE_ID), and the file table describes the agent reading task_id. Implements TC-6232 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
…s not missed The pre_script's JQL search used a single page (<=50 results), so an exact PR-linked issue falling beyond the first 50 broad `~` recall results was dropped and resolve-gated-issue emitted a false ADR-0072 skip even though a unique gated issue existed. Add search_jql_all() to jira-client.py: it follows the enhanced endpoint's nextPageToken cursor across every page, aggregates the issues into one response, and fails loud (exit 1) if a runaway server never stops offering a cursor rather than truncating silently. Expose it via a `search_jql --all` flag and switch pre-verify-pr.sh's gating search to `--all`. The Python exact-match verification in resolve_gated_issue remains the source of truth. Tests: search_jql_all collects issues beyond page 1, stops without a token, and guards against runaway pagination; resolve_gated_issue resolves an exact match at index 55 of a 60-issue aggregated result; and a shell-source guard asserts the gating search uses --all. Implements TC-6233 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
…iting sandbox input The status and label gates were checked on the lightweight JQL search response, but the full issue used to write verify-pr-input.json is fetched in a second request. If the issue left Review, lost the ai-generated-jira label, or had its Git Pull Request field changed between the two, the stale successful gate still launched the verification (TOCTOU). Re-run the exact PR-URL + status + label gate on the full issue immediately before the write via a new revalidate-gate command (reuses resolve_gated_issue so the acceptance rule stays single-sourced). A failure maps to the same ADR-0072 skip as the initial gate. Implements TC-6234 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
Re-pin the base to the commit carrying the TC-6232/6233/6234 script fixes and re-lock so every transitive child resolves at the new SHA. Without this the fullsend run path (local + CI) serves the pre-edit pinned content and never exercises the fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
…tage Add privilege_levels to harness/verify-pr.yaml mapping run-stages to the review role's named mint levels (fullsend ADR-0073): pre_script and runtime get read, post_script keeps write. pre_script only reads GitHub, the sandbox has no api.github.com egress (split-trust I/O), and only post_script writes PR review comments/replies/report. Applies least-privilege now and survives fullsend #6516 flipping the mint default from write to read. Implements TC-6243 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
harness/verify-pr.yaml gained privilege_levels; re-pin the .fullsend base to a9099de (sha256 3c9dc221) and re-lock so local and CI resolve the new bytes. Implements TC-6243 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
The push/pull_request branch filters still named the old feature branch TC-5787-verify-pr-fullsend-v037, which was renamed to verify-pr-fullsend (TC-6180). The stale name matched no live branch, so the pytest suite never ran as a check on PRs targeting the feature branch (e.g. #298). Replace it with verify-pr-fullsend in both filters and comments; keep the TC-5816 note to drop the non-main entry after the feature branch merges to main. Implements TC-6244 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
Add verify-pr-fullsend to the push/pull_request branch filters of both workflows so plugin validation and skill lint run on sub-task PRs into the feature branch, matching python-tests.yml (TC-6244). Entries carry a TC-5816 cleanup note to be removed when the feature branch merges to main. Implements TC-6245 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
feat(verify-pr): derive + gate Jira key from PR URL via JQL in pre_script (TC-6190)
|
🤖 Finished Verify Pr · ⏭️ Skipped (no Jira issue links PR #299 in its Git Pull Request field) · Started 2:29 PM UTC · Completed 2:29 PM UTC Commit: Effort: high |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Eval Results
Eval Results: verify-pr
| Eval | Passed | Failed | Pass Rate |
|---|---|---|---|
| eval-1 | 13/13 | 0 | 100% |
| eval-2 | 11/11 | 0 | 100% |
| eval-3 | 17/17 | 0 | 100% |
| eval-4 | 10/10 | 0 | 100% |
| eval-5 | 10/10 | 0 | 100% |
| eval-6 | 9/10 | 1 | 90% |
Failed Assertions
eval-6: 1 failing assertion
- Assertion: "The human reviewer comment from reviewer-b (comment id 50001) is NOT misidentified as an eval result — it is processed as a normal review comment (classified in review-50001.md) and its classification does not reference eval detection or eval metrics"
Evidence: "review-50001.md correctly classifies the comment as 'Code Change Request' (not an eval result), and the file exists as review-50001.md confirming normal review processing. However, the classification DOES reference eval detection: line 6 says 'Type: Inline review comment (NOT an eval result)' and reasoning point 5 (lines 28-30) explicitly discusses eval detection markers: 'Not an eval result: This comment is from a human reviewer (reviewer-b), not from github-actions[bot]. It does not contain "## Eval Results" or "sdlc-workflow/run-evals" markers.' The assertion requires the classification does not reference eval detection, but it clearly does."
Pass rate: 98% · Tokens: 89,093 · Duration: 802s
Generated by sdlc-workflow/run-evals v0.13.9
Add fullsend.md §Acceptance run → "E2E acceptance via CI dispatch (TC-6192)" documenting the CI-gated verify-pr dispatch proven end to end on PR #300 / TC-6254: qualifying PR triggers the workflow, wait-for-checks releases after CI reaches terminal, verify-pr posts to both the PR and Jira. Records all four acceptance criteria — CI-pass run (python-tests success → Overall WARN), CI-fail run (python-tests failure still releases the wait → CI Status FAIL, sub-task auto-created), report on both surfaces, and idempotency (re-run same head, no duplicate reports). Notes DENIED egress = split-trust policy working, and the interim personal-token Jira authorship pending the SA (TC-6191). Implements TC-6192 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The acceptance results table lists five criteria; the lead-in sentence said "four". Corrects the count to match (Sourcery nit surfaced by the verify-pr WARN on PR #301). Implements TC-6192 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 Finished Verify Pr · ⏭️ Skipped (no Jira issue links PR #299 in its Git Pull Request field) · Started 7:36 AM UTC · Completed 7:36 AM UTC Commit: Effort: high |
In the CI-dispatched (sandbox) verify-pr path the CI Status check had no data — no gh CLI or egress in the sandbox — so it degraded to WARN "CI status could not be retrieved" (PR #301, TC-6192). Extend the split-trust prefetch so the pre_script fetches the PR head-SHA CI check-run outcomes on the trusted runner and embeds them in the sandbox bundle; correctness.md Check 1 reads them in sandbox mode and keeps the `gh` on-demand path for interactive mode. - pre-verify-pr.sh: prefetch repos/<repo>/commits/<sha>/check-runs into check-runs.json, reduced to name/status/conclusion/details_url - pre_verify_pr.py: build_github_bundle embeds github.check_runs (defaults to []); _github_from_dir reads check-runs.json - verify-pr-input.schema.json: add required github.check_runs (github is additionalProperties:false, so the new key must be declared) - correctness.md: Check 1 sandbox-mode branch reads pre-fetched check-runs; 1b notes failure logs are unavailable in the sandbox - dispatch-template.md, SKILL.md Step 0.7: document github.check_runs - test_pre_verify_pr.py: cover check_runs in the bundle and transform E2E re-verification on a live dispatched PR (PASS on green, FAIL on red) is deferred to a CI run. Implements TC-6257 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
The CI-status prefetch gives the sandbox check-run outcomes, but on a FAIL correctness.md Check 1b still had no failure logs to analyse: the sandbox has no gh CLI or egress to run `gh run view --log-failed`, so it inferred the what/why/fix from the diff alone. Extend the split-trust prefetch to capture the logs too. The pre_script (on the trusted runner, where GH_TOKEN lives) extracts the distinct GitHub Actions run IDs from each FAILED check-run's details_url and concatenates their `gh run view --log-failed` output into a single check-run-logs.txt, mounted read-only beside verify-pr-input.json. host_files mounts single files only and the failed set is dynamic, so one combined file is used rather than per-check paths. The file is always created (empty when nothing failed) so its mount is never missing; the bundle carries only its sandbox path (github.check_run_logs_path, "" when empty), never the log text — so the large content stays out of the schema and off the sub-agent's context until Check 1b Reads it on a FAIL. The wait-for-checks job (fullsend-verify-pr.yml) guarantees terminal conclusions before dispatch, so a failed check's log is complete and fetchable at that point. Non-Actions checks (no actions/runs URL) are skipped; 1b falls back to the diff + details_url for those. Interactive mode is unchanged (fetches via gh on demand). - pre-verify-pr.sh: extract failed Actions run IDs (deduped) and pack their --log-failed output into check-run-logs.txt; always create the file - pre_verify_pr.py: build_github_bundle gains check_run_logs_path (default ""); _github_from_dir emits the mounted sandbox path when the log file is non-empty - verify-pr-input.schema.json: add required github.check_run_logs_path (string) - .fullsend/harness/verify-pr.yaml: mount check-run-logs.txt (optional) - correctness.md / dispatch-template.md / SKILL.md: Check 1b reads the prefetched log file via the CI Failure Logs input on a FAIL; document the new key - test_pre_verify_pr.py: cover the new bundle field and the path derivation (present/empty/absent) TC-6257 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The split-trust prefetch children (pre-verify-pr.sh, correctness.md, SKILL.md, dispatch-template.md, verify-pr-input.schema.json) added in 14b8be6/fa3f4b7d only take effect in a CI-dispatched run when the URL-pinned harness base advances to a commit that contains them. The base file bytes are unchanged (sha256 3c9dc221…), so only the pinned commit moves a9099de → fa3f4b7; `fullsend lock` refreezes all 11 children (notably pre-verify-pr.sh) at fa3f4b7. Without this the sandbox keeps running the pre-TC-6257 pinned skill, whose bundle carries no github.check_runs — CI Status falls back to the WARN "unavailable in sandbox" path instead of sourcing prefetched data. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI Status is now sourced from prefetched github.check_runs, and on a CI failure Correctness Check 1b reads the prefetched check-run-logs.txt. Proven live via CI dispatch on vehicle PR #303 / Jira TC-6266: - post-re-pin run 35075966985 sourced CI Status from prefetched check_runs (data the pre-TC-6257 sandbox could not obtain); - vehicle-only red commit 8865e11 (canary test) → verify-pr 35077404421 reported CI Status FAIL and Check 1b surfaced the runtime-computed, log-only canary cc08a78a24bc66f6 (never a literal in the diff), auto-creating CI-failure sub-task TC-6271. Also documents that the harness base re-pin is load-bearing (the pinned children must advance to a commit containing the prefetch code) and supersedes the TC-6192 "CI status unavailable in sandbox" caveat. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat(verify-pr): prefetch CI check-run outcomes into sandbox input
|
🤖 Finished Verify Pr · ⏭️ Skipped (no Jira issue links PR #299 in its Git Pull Request field) · Started 10:27 AM UTC · Completed 10:27 AM UTC Commit: Effort: high |
…cedures Add a CI deployment section (per-repo fullsend github setup / inference provision / org GitHub App install), a dispatch-design section (inline pull_request wait via lewagon/wait-on-check-action, CEL trigger, pre_script Jira gating), a day-2 variables/secrets management table, and maintenance procedures for bumping the pinned fullsend version and keeping the two plugin-version files in sync. Refresh stale v0.37.0 references. Implements TC-6193 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 Finished Verify Pr · ⏭️ Skipped (no Jira issue links PR #299 in its Git Pull Request field) · Started 10:52 AM UTC · Completed 10:52 AM UTC Commit: Effort: high |
…o-test Switch fullsend-verify-pr.yml from pull_request to pull_request_target so fork-origin PRs get the upstream vars/secrets/OIDC the mint step needs (fixes "FULLSEND_MINT_URL is not set"; fullsend ADR-0009). The workflow still never checks out or executes fork code — it builds the dispatch matrix from github.event alone and hands off to reusable-dispatch.yml. Add an ok-to-test maintainer-label gate as defense-in-depth: fork PRs dispatch only after a maintainer applies the label; the remove-ok-to-test job strips it on every new push (synchronize) to force re-review. Same-repo PRs are unchanged. Update fullsend.md with the pull_request_target + ok-to-test model and remove the obsolete "upstream head branch required" framing. Implements TC-6331 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The head-SHA check-run prefetch enumerated every check-run on the commit, including verify-pr's own workflow (`fullsend verify-pr`) runs — the in-progress dispatch and any superseded prior attempt. Those are non-terminal/failed at evaluation time, so correctness.md Check 1a mapped them to pending/failed and forced a permanent self-referential CI Status WARN/FAIL: verify-pr could never report CI Status = PASS (TC-6332). Add the CI-Status analogue of Step 1's `running-workflow-name` self- exclusion. pre-verify-pr.sh enumerates this workflow's own check-run names for the head SHA — across all of its runs at that SHA, so superseded attempts are covered — and passes them to a pure, unit-testable Python helper `filter_own_check_runs`, which drops them before they reach `github.check_runs`. Exclusion is by workflow NAME, not run ID. Name gathering is best-effort: a token lacking actions:read degrades to no self-exclusion (prior behavior) with a warning rather than aborting. Fixes TC-6332. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reword correctness.md Check 1a sandbox-mode note: the own-check-run self-exclusion is best-effort (needs actions:read) and degrades to a no-op with a WARNING when enumeration fails, so the CI Status input can still contain verify-pr's own runs. Instruct the evaluator to map each entry on its merits and note own-run-only non-passing entries in the evidence rather than treating the PR as blocked. Implements TC-6349 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TC-6343 Option A self-pin: move the base URL commit and re-lock so the verify-pr CI run fetches the fixed pre-verify-pr.sh (CI-Status own-workflow self-exclusion) and the reworded correctness.md Check 1a from this PR's own fix commits, rather than the stale fa3f4b7 pin. - .fullsend/harness/verify-pr.yaml: base URL commit fa3f4b7 -> da053ba (root harness/verify-pr.yaml is byte-identical, sha256 unchanged) - .fullsend/lock.yaml: fullsend lock regenerates all 11 child pins at da053ba; pre-verify-pr.sh, pre_verify_pr.py, and correctness.md hashes now reflect the TC-6343 / TC-6349 fixes Merge via merge commit (squash disabled) so da053ba stays reachable and the pin keeps resolving after merge. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(verify-pr): self-exclude own workflow check-runs from CI Status
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — do not merge. Continuous-CI vehicle for the
verify-pr-fullsendfeature branch (TC-6180 CI-deployment feature).Repo CI (validate-plugins, skillsaw, and the path-filtered evals) triggers only on
pull_request/pushtomain. This draft PR gives the feature branch continuous CI againstmainas sub-tasks land — including the expensive, main-only evals that are intentionally not run on per-sub-task PRs into the feature branch.Sub-tasks continue to PR into
verify-pr-fullsendindividually (e.g. #298). This PR is the aggregate gate; it stays draft until the feature is complete and TC-5816 (merge bookend) removes the temporary feature-branch CI entries.🤖 Generated with Claude Code