Conversation
In the CI-dispatched (sandbox) verify-pr path the CI Status check had no data — no gh CLI or egress in the sandbox — so it degraded to WARN "CI status could not be retrieved" (PR #301, TC-6192). Extend the split-trust prefetch so the pre_script fetches the PR head-SHA CI check-run outcomes on the trusted runner and embeds them in the sandbox bundle; correctness.md Check 1 reads them in sandbox mode and keeps the `gh` on-demand path for interactive mode. - pre-verify-pr.sh: prefetch repos/<repo>/commits/<sha>/check-runs into check-runs.json, reduced to name/status/conclusion/details_url - pre_verify_pr.py: build_github_bundle embeds github.check_runs (defaults to []); _github_from_dir reads check-runs.json - verify-pr-input.schema.json: add required github.check_runs (github is additionalProperties:false, so the new key must be declared) - correctness.md: Check 1 sandbox-mode branch reads pre-fetched check-runs; 1b notes failure logs are unavailable in the sandbox - dispatch-template.md, SKILL.md Step 0.7: document github.check_runs - test_pre_verify_pr.py: cover check_runs in the bundle and transform E2E re-verification on a live dispatched PR (PASS on green, FAIL on red) is deferred to a CI run. Implements TC-6257 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code
The CI-status prefetch gives the sandbox check-run outcomes, but on a FAIL correctness.md Check 1b still had no failure logs to analyse: the sandbox has no gh CLI or egress to run `gh run view --log-failed`, so it inferred the what/why/fix from the diff alone. Extend the split-trust prefetch to capture the logs too. The pre_script (on the trusted runner, where GH_TOKEN lives) extracts the distinct GitHub Actions run IDs from each FAILED check-run's details_url and concatenates their `gh run view --log-failed` output into a single check-run-logs.txt, mounted read-only beside verify-pr-input.json. host_files mounts single files only and the failed set is dynamic, so one combined file is used rather than per-check paths. The file is always created (empty when nothing failed) so its mount is never missing; the bundle carries only its sandbox path (github.check_run_logs_path, "" when empty), never the log text — so the large content stays out of the schema and off the sub-agent's context until Check 1b Reads it on a FAIL. The wait-for-checks job (fullsend-verify-pr.yml) guarantees terminal conclusions before dispatch, so a failed check's log is complete and fetchable at that point. Non-Actions checks (no actions/runs URL) are skipped; 1b falls back to the diff + details_url for those. Interactive mode is unchanged (fetches via gh on demand). - pre-verify-pr.sh: extract failed Actions run IDs (deduped) and pack their --log-failed output into check-run-logs.txt; always create the file - pre_verify_pr.py: build_github_bundle gains check_run_logs_path (default ""); _github_from_dir emits the mounted sandbox path when the log file is non-empty - verify-pr-input.schema.json: add required github.check_run_logs_path (string) - .fullsend/harness/verify-pr.yaml: mount check-run-logs.txt (optional) - correctness.md / dispatch-template.md / SKILL.md: Check 1b reads the prefetched log file via the CI Failure Logs input on a FAIL; document the new key - test_pre_verify_pr.py: cover the new bundle field and the path derivation (present/empty/absent) TC-6257 Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer's GuideThis throwaway E2E vehicle PR moves sandbox verify-pr CI validation from on-demand GitHub CLI calls to trusted-runner-prefetched head-SHA check runs, with failed logs mounted separately and read only when needed, while retaining interactive-mode behavior and adding transformation/unit coverage. Sequence diagram for prefetched verify-pr CI validationsequenceDiagram
participant Runner as Trusted Runner
participant GitHub as GitHub API
participant PreScript as pre-verify-pr.sh
participant Bundle as verify-pr-input.json
participant Sandbox as Verify-pr Sandbox
participant Correctness as Correctness Sub-agent
Runner->>PreScript: Run pre-verify-pr.sh
PreScript->>GitHub: Fetch check-runs for COMMIT_SHA
GitHub-->>PreScript: check-runs.json
PreScript->>GitHub: gh run view --log-failed for failed runs
GitHub-->>PreScript: check-run-logs.txt
PreScript->>Bundle: Embed check_runs and check_run_logs_path
Runner->>Sandbox: Mount bundle and check-run-logs.txt
Sandbox->>Correctness: Provide CI Status from github.check_runs
Correctness-->>Sandbox: PASS, WARN, or FAIL
opt CI status is FAIL
Correctness->>Sandbox: Read CI Failure Logs path
Sandbox-->>Correctness: Concatenated failed-check logs
Correctness->>Correctness: Emit create-sub-task with details_url
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
🤖 Finished Verify Pr · ✅ Success · Started 8:36 AM UTC · Completed 8:46 AM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $5.35 |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="plugins/sdlc-workflow/scripts/pre-verify-pr.sh" line_range="212-215" />
<code_context>
+: > "${CHECK_LOGS_FILE}"
+FAILED_RUN_IDS=$(jq -r '
+ [ .[]
+ | select(.conclusion // "" | IN("failure", "timed_out", "cancelled", "action_required"))
+ | (.details_url // "")
+ | select(test("actions/runs/[0-9]+"))
+ | capture("actions/runs/(?<id>[0-9]+)").id
+ ] | unique | .[]' "${PRE_OUTPUT_DIR}/check-runs.json")
+for run_id in ${FAILED_RUN_IDS}; do
</code_context>
<issue_to_address>
**issue (bug_risk):** The prefetch and sandbox CI-status mapping omit terminal GitHub conclusions `startup_failure` and `stale`. Those checks are neither selected for failure-log collection nor classified as failed by Check 1, so a red CI run can be reported without a FAIL verdict.
**Triggers:** When a head-SHA check run ends with `startup_failure` or `stale`.
**Suggested fix:** Treat `startup_failure` and `stale` as failed conclusions in both the shell selector and the Check 1 mapping.
</issue_to_address>
### Comment 2
<location path="plugins/sdlc-workflow/scripts/pre-verify-pr.sh" line_range="218-223" />
<code_context>
+ ] | unique | .[]' "${PRE_OUTPUT_DIR}/check-runs.json")
+for run_id in ${FAILED_RUN_IDS}; do
+ {
+ echo "===== CI run ${run_id} — failed steps ====="
+ gh run view "${run_id}" --log-failed -R "${PR_REPO}" 2>&1 \
+ || echo "(log fetch failed for run ${run_id}; see its details_url in check_runs)"
+ echo
+ } >> "${CHECK_LOGS_FILE}"
+done
</code_context>
<issue_to_address>
**issue (bug_risk):** When `gh run view --log-failed` fails, the shell still writes a header and a failure note, making `check_run_logs_path` non-empty. Check 1b then reads that file as if it were the failure log and does not take its documented diff/details_url fallback, so the actual CI failure is analyzed without its available fallback evidence.
**Triggers:** When an Actions run is failed but its log fetch returns a non-zero status.
**Suggested fix:** Only expose `check_run_logs_path` when at least one log fetch succeeds, or mark failed fetches so Check 1b explicitly falls back to the diff and check-run metadata.
```suggestion
run_log=$(mktemp)
if gh run view "${run_id}" --log-failed -R "${PR_REPO}" > "${run_log}" 2>&1; then
{
echo "===== CI run ${run_id} — failed steps ====="
cat "${run_log}"
echo
} >> "${CHECK_LOGS_FILE}"
else
echo "(log fetch failed for run ${run_id}; see its details_url in check_runs)" >&2
fi
rm -f "${run_log}"
```
</issue_to_address>Sourcery assessment
Needs a human reviewer. 2 findings to address first, and if the prefetched check-run data or failure-log path is wrong, the verifier could report an incorrect CI result and create an erroneous Jira sub-task, or miss a needed one. Reverting prevents future misclassification, but any sub-task or review decision already created would remain and require cleanup.
Blocking findings: plugins/sdlc-workflow/scripts/pre-verify-pr.sh:215, plugins/sdlc-workflow/scripts/pre-verify-pr.sh:223
| | select(.conclusion // "" | IN("failure", "timed_out", "cancelled", "action_required")) | ||
| | (.details_url // "") | ||
| | select(test("actions/runs/[0-9]+")) | ||
| | capture("actions/runs/(?<id>[0-9]+)").id |
There was a problem hiding this comment.
issue (bug_risk): The prefetch and sandbox CI-status mapping omit terminal GitHub conclusions startup_failure and stale. Those checks are neither selected for failure-log collection nor classified as failed by Check 1, so a red CI run can be reported without a FAIL verdict.
Triggers: When a head-SHA check run ends with startup_failure or stale.
Suggested fix: Treat startup_failure and stale as failed conclusions in both the shell selector and the Check 1 mapping.
There was a problem hiding this comment.
[sdlc-workflow/verify-pr] Classified as code change request — sub-task TC-6268 created to address this feedback (treat startup_failure/stale as failed CI conclusions in the prefetch selector and Check 1 mapping).
| { | ||
| echo "===== CI run ${run_id} — failed steps =====" | ||
| gh run view "${run_id}" --log-failed -R "${PR_REPO}" 2>&1 \ | ||
| || echo "(log fetch failed for run ${run_id}; see its details_url in check_runs)" | ||
| echo | ||
| } >> "${CHECK_LOGS_FILE}" |
There was a problem hiding this comment.
issue (bug_risk): When gh run view --log-failed fails, the shell still writes a header and a failure note, making check_run_logs_path non-empty. Check 1b then reads that file as if it were the failure log and does not take its documented diff/details_url fallback, so the actual CI failure is analyzed without its available fallback evidence.
Triggers: When an Actions run is failed but its log fetch returns a non-zero status.
Suggested fix: Only expose check_run_logs_path when at least one log fetch succeeds, or mark failed fetches so Check 1b explicitly falls back to the diff and check-run metadata.
| { | |
| echo "===== CI run ${run_id} — failed steps =====" | |
| gh run view "${run_id}" --log-failed -R "${PR_REPO}" 2>&1 \ | |
| || echo "(log fetch failed for run ${run_id}; see its details_url in check_runs)" | |
| echo | |
| } >> "${CHECK_LOGS_FILE}" | |
| run_log=$(mktemp) | |
| if gh run view "${run_id}" --log-failed -R "${PR_REPO}" > "${run_log}" 2>&1; then | |
| { | |
| echo "===== CI run ${run_id} — failed steps =====" | |
| cat "${run_log}" | |
| echo | |
| } >> "${CHECK_LOGS_FILE}" | |
| else | |
| echo "(log fetch failed for run ${run_id}; see its details_url in check_runs)" >&2 | |
| fi | |
| rm -f "${run_log}" |
There was a problem hiding this comment.
[sdlc-workflow/verify-pr] Classified as code change request — sub-task TC-6269 created to address this feedback (only expose check_run_logs_path when a --log-failed fetch succeeds, else fall back in Check 1b).
Verification Report for TC-6266 (commit fa3f4b7)
Overall: FAILThe only failing check is Commit Traceability. This run verifies the throwaway E2E vehicle task TC-6266, but both PR commits carry the underlying feature ID TC-6257 in their trailers ( Non-blocking observations (informational rows do not affect the overall result):
No sub-tasks were created and no review feedback was present. This report is informational — a human reviewer decides whether to merge. This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9. |
The split-trust prefetch children (pre-verify-pr.sh, correctness.md, SKILL.md, dispatch-template.md, verify-pr-input.schema.json) added in 14b8be6/fa3f4b7d only take effect in a CI-dispatched run when the URL-pinned harness base advances to a commit that contains them. The base file bytes are unchanged (sha256 3c9dc221…), so only the pinned commit moves a9099de → fa3f4b7; `fullsend lock` refreezes all 11 children (notably pre-verify-pr.sh) at fa3f4b7. Without this the sandbox keeps running the pre-TC-6257 pinned skill, whose bundle carries no github.check_runs — CI Status falls back to the WARN "unavailable in sandbox" path instead of sourcing prefetched data. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 Finished Verify Pr · ✅ Success · Started 8:51 AM UTC · Completed 9:02 AM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $5.66 |
|
[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review — Classified as code change request — this review body restates two inline bug_risk findings that are already tracked as sub-tasks TC-6268 (startup_failure/stale conclusion handling) and TC-6269 (check_run_logs_path exposure). No additional sub-task created — it duplicates the inline comment threads. |
Verification Report for TC-6266 (commit 1185ae2)
Overall: FAILOverall: FAIL — two hard-fail checks. CI Status FAIL: the head-SHA check This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9. |
Deliberate failing pytest whose root cause (a runtime-computed 16-hex canary) appears ONLY in the CI failure log, never as a literal in the diff. Exercises the CI-dispatched verify-pr Check 1b: CI Status must be sourced from the prefetched github.check_runs and the failure analysed from the prefetched check-run-logs.txt. Vehicle-only — never merged. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 Finished Verify Pr · ✅ Success · Started 9:07 AM UTC · Completed 9:16 AM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $4.80 |
Verification Report for TC-6266 (commit 8865e11)
Overall: FAIL This is the intentional red commit of the TC-6257 live E2E acceptance vehicle. Two checks fail as expected/observed:
Review feedback: 2 Sourcery This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9. |
CI Status is now sourced from prefetched github.check_runs, and on a CI failure Correctness Check 1b reads the prefetched check-run-logs.txt. Proven live via CI dispatch on vehicle PR RHEcosystemAppEng#303 / Jira TC-6266: - post-re-pin run 35075966985 sourced CI Status from prefetched check_runs (data the pre-TC-6257 sandbox could not obtain); - vehicle-only red commit 8865e11 (canary test) → verify-pr 35077404421 reported CI Status FAIL and Check 1b surfaced the runtime-computed, log-only canary cc08a78a24bc66f6 (never a literal in the diff), auto-creating CI-failure sub-task TC-6271. Also documents that the harness base re-pin is load-bearing (the pinned children must advance to a commit containing the prefetch code) and supersedes the TC-6192 "CI status unavailable in sandbox" caveat. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
E2E acceptance for TC-6257 captured in fullsend.md (PR #302). CI-status prefetch + Check 1b log-read proven live (canary cc08a78a24bc66f6, CI-failure sub-task TC-6271). Closing throwaway vehicle. |
| # and details_url for the failure-log link) to keep the bundle small. A PR with no | ||
| # checks yields an empty array — consistent with the reviews/comments empties. | ||
| gh api --paginate --slurp "repos/${PR_REPO}/commits/${COMMIT_SHA}/check-runs" \ | ||
| | jq '[.[].check_runs[] | {name, status, conclusion, details_url}]' > "${PRE_OUTPUT_DIR}/check-runs.json" |
There was a problem hiding this comment.
wouldn't a red CI come out as WARN and 1b would never run ?
2 participants
Throwaway vehicle PR for the TC-6257 live E2E acceptance run (CI-dispatched verify-pr).
Delete after evidence is captured in fullsend.md.
🤖 Generated with Claude Code
Summary by Sourcery
Validate sandboxed verify-pr CI assessment by prefetched check-run results and failure logs.
New Features:
Bug Fixes:
Enhancements:
CI:
Documentation:
Tests:
Chores: