Skip to content

skills: read FNALbuild's table before verifying anything yourself - #15

Open
oksuzian wants to merge 1 commit into
mainfrom
skills-ci-coverage
Open

skills: read FNALbuild's table before verifying anything yourself#15
oksuzian wants to merge 1 commit into
mainfrom
skills-ci-coverage

Conversation

@oksuzian

Copy link
Copy Markdown
Contributor

Adds a note telling the review skill not to verify things CI already verifies.

Why

mu2e/buildtest is not one test — it is about twenty, and most have no
separate commit-status context
. The statuses API returns roughly eleven
contexts; the rest appear only as rows in FNALbuild's result table. Reading
statuses alone reports that a check does not exist when it ran and passed.

Two of the invisible ones matter a lot for geometry PRs:

  • g4surfaceCheck — runs Offline/Mu2eG4/fcl/surfaceCheck.fcl, the Geant4
    overlap check over the whole geometry. It is graded on log content rather
    than exit code (overlaps are G4 warnings, so the job would otherwise exit 0),
    with a LEGAL > 0 guard against a vacuous pass.
  • rootOverlaps — an independent overlap check via ROOT.

What happened

Reviewing Offline#1933, a building-geometry vertex move, the skill concluded
from the statuses API that CI does not check overlaps. It then wrote its own
Python analysis of the hall geometry and asked the author to run
geom_SurfaceCheck.txt by hand. Both g4surfaceCheck and rootOverlaps were
already green at that head, so the request was redundant work for the author.

The home-made scan was also wrong in coverage while reading as verified:
Mu2eHallMaker loads four families of solids, and the scan matched only
building.* prefixes — silently dropping 100 dirt, 13 rotated and 3 trap
volumes. It still returned "no overlaps", which is indistinguishable from a
real result. The review shipped a 🟢 claiming "every active hall volume".

Changes

  • New "What buildtest already covers — read the table, not the statuses"
    section under Triggering a CI Build: lists what runs inside buildtest, says
    not to re-derive it and never to ask an author to run one of those checks,
    and says that an independent check is a cross-check of CI, never the
    evidence. Points at Mu2e/codetools for the job lists and Mu2e/CI for the
    test vocabulary.
  • New Evidence Rules bullet: find out what CI already verified before
    verifying anything yourself.

Net +42 lines, no other content touched. This generalises the rule the skill
already had for the whitespace check, which runs inside buildtest for the same
reason.

🤖 Generated with Claude Code

mu2e/buildtest runs ~20 checks, most with no separate commit-status
context. Reading the statuses API alone reports that a check does not
exist when it ran and passed -- g4surfaceCheck and rootOverlaps are the
cases that matter, since they cover geometry overlaps across the whole
geometry.

Adds a "What buildtest already covers" section listing what runs inside
buildtest, and an Evidence Rules bullet telling the reviewer to find out
what CI verified before verifying it themselves. Both say plainly not to
hand-roll a substitute and not to ask an author to run a check CI ran.

Prompted by a review that asked the author to run surfaceCheck.fcl when
CI had already run it green, backed by a home-made scan that silently
covered one of the four solid families the geometry loads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rlcee

rlcee commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Can't the AI PR system pull previous comments? Why is the CI results table hidden from it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants