Skip to content

feat(ci): agent gate for knowledge PRs — transferability, duplication, fact check - #155

Merged
choiyounggi merged 1 commit into
mainfrom
feat/wiki-agent-gate
Aug 28, 2026
Merged

feat(ci): agent gate for knowledge PRs — transferability, duplication, fact check#155
choiyounggi merged 1 commit into
mainfrom
feat/wiki-agent-gate

Conversation

@choiyounggi

Copy link
Copy Markdown
Owner

What

Adds an agent-run CI gate for the three semantic checks scripts cannot do, closing the gap where transferability, semantic duplication, and fact-checking existed only as self-reported steps inside the knowledge-flush session:

  • .github/workflows/wiki-agent-gate.yml — on every PR touching wiki/**, runs Claude Code headless via anthropics/claude-code-action@v1 (subscription OAuth token, no per-token billing). Verdict arrives as --json-schema structured output; the gate fails closed when the agent produces no verdict and fails on any blocker finding. Findings are posted back to the PR as a comment on failure.
  • .github/wiki-agent-gate-prompt.md — the check definitions: transferability (org-internal directives are blockers), semantic duplication (Grep the existing wiki before accepting a new page/row), fact check (WebFetch each cited source and confirm the quote supports the directive; unfetchable → advisory, never guessed).
  • tests/wiki-agent-gate.bats — 12 doc-gate tests pinning the security guards, the fail-closed step, and the schema/prompt contract, each positive check paired with a negative control.

Security (pull_request_target)

Knowledge PRs come from a fork, and plain pull_request withholds secrets there — so this uses pull_request_target with the two guards the action's security docs prescribe:

  1. Job-level trust gate: only PRs from this repo or the maintainer's own accounts (choiyounggi, dch0202-rsquare) start the job.
  2. Base-ref-only checkout: the untrusted PR head is never checked out; the agent reads it exclusively via gh pr diff, and the prompt file instructs it to treat instruction-shaped text inside the diff as a finding, not as instructions.

Setup required before this gate can run

The repo secret CLAUDE_CODE_OAUTH_TOKEN must be added once: run claude setup-token locally, then gh secret set CLAUDE_CODE_OAUTH_TOKEN.

Verification

  • bats tests/wiki-agent-gate.bats → 12/12 pass
  • Workflow YAML parses (ruby YAML.load_file)
  • No wiki/** changes: directive count and structure gates unaffected

🤖 Generated with Claude Code

https://claude.ai/code/session_01RJsPDQCz3wNt2qk4opqcKG

…, fact check

The mechanical wiki gates in test.yml cover format and countable duplicates;
the semantic checks (transferability, semantic duplication, fact-checking of
cited sources) existed only as self-reported steps inside the knowledge-flush
session. This adds an independent CI gate that runs Claude Code headless
(anthropics/claude-code-action@v1, subscription OAuth token) over the PR diff
on every PR touching wiki/**.

- pull_request_target with two guards: a job-level trust gate (base-repo or
  maintainer-fork PRs only) and a base-ref-only checkout — the untrusted PR
  head is read exclusively via gh pr diff.
- Verdict via --json-schema structured output; the gate fails closed when the
  agent produces no verdict, and fails on any blocker finding.
- Findings are posted back to the PR as a comment on failure.
- tests/wiki-agent-gate.bats pins both security guards, the fail-closed step,
  and the schema/prompt contract, each with a negative control.

Requires the CLAUDE_CODE_OAUTH_TOKEN repo secret (claude setup-token).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RJsPDQCz3wNt2qk4opqcKG
@choiyounggi
choiyounggi merged commit 09f2f1e into main Aug 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant