CI: make the Gemini PR reviewer actually review - #1140
Merged
Conversation
The job has never posted a review. It passed the Gemini API key but never
set a model, so PR-Agent kept its OpenAI defaults (`gpt-5.6`, falling back
to `gpt-5.6-terra`) and every call failed with
AuthenticationError - Incorrect API key provided: dummy_key
PR-Agent swallows that error, so the run still went green and the failure
was invisible. Setting `config.model` / `config.fallback_models` to Gemini
models is the actual fix.
Three other reasons it never worked:
- `pull_request` gives no secrets to fork PRs, and most contributions are
fork PRs (23 of the 27 currently open). Those runs sat in `action_required`
(22 of the last 60) and would have had an empty key even once approved.
`pull_request_target` is safe here because the job never checks the PR out.
- `synchronize` is not in PR-Agent's `pr_actions`, so every push logged
"Skipping action: synchronize" after a ~30 s image build - 38 of the last
60 runs were that no-op.
- `issue_comment` runs the workflow from the default branch, and this file
only exists on `develop`, so `/review` and `/ask` never fired once: all
74 recorded runs were `pull_request`.
Also raise the token ceiling off the 32k default, since Gemini takes 1M and
larger diffs were being truncated; leave the human-written PR description
alone by turning `auto_describe` off; and drop the unused `contents: write`.
The action ref moves from `@main` to a release tag. Note that this pins the
action definition only: `Dockerfile.github_action_dockerhub` is a one-liner
that pulls `pragent/pr-agent:github_action`, a floating tag, so the agent
itself is still whatever that image currently holds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1140 +/- ##
===========================================
+ Coverage 82.18% 82.80% +0.62%
===========================================
Files 122 129 +7
Lines 16355 16812 +457
===========================================
+ Hits 13441 13922 +481
+ Misses 2914 2890 -24 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request type
Checklist
CHANGELOG.md— no action needed; and per the file's own header, GitHub maintenance does not belong thereCurrent behavior
The Gemini PR reviewer has never posted a review on a single PR, and it never went red either, which is why it went unnoticed for so long. There are four independent problems.
1. The actual failure: no model was ever selected.
The workflow passed
GOOGLE_AI_STUDIO.GEMINI_API_KEYbut never setconfig.model, so PR-Agent kept its OpenAI defaults (gpt-5.6, falling back togpt-5.6-terra). With no OpenAI key present, LiteLLM used a placeholder and every one of the three tools died:PR-Agent catches that exception rather than propagating it, so the job still reported success. Example: run 31350336071.
2. Fork PRs could never work. A
pull_requestrun triggered from a fork gets no secrets, so the key would have been empty regardless — and those runs did not even start, sitting inaction_requiredawaiting approval (22 of the last 60). This is the majority of our traffic: 23 of the 27 currently open PRs come from forks.3.
synchronizewas pure waste. It is not in PR-Agent'spr_actions, andhandle_push_triggerdefaults to off, so every push loggedSkipping action: synchronizeafter spending ~30 s building the container. 38 of the last 60 runs were that no-op.4. Comment commands never fired once.
issue_commentalways runs the workflow from the default branch (master), and this file only exists ondevelop— so/review,/describeand/askwere dead on arrival. Confirmed empirically: all 74 recorded runs of this workflow arepull_request, zero areissue_comment.New behavior
config.model/config.fallback_modelsset to Gemini (gemini/gemini-3.5-pro, falling back togemini/gemini-3.6-flash). This is the fix for thedummy_keyfailure.pull_request→pull_request_target, so fork PRs get the key and run without manual approval. This is safe here because the job never checks the pull request out: PR-Agent reads the diff over the GitHub API, and the only step is a container action, so no untrusted code is executed.synchronizedropped from the trigger list.github_action_config.handle_push_triggeris the knob if we ever want a re-review on each push.config.max_model_tokensraised off the 32k default. Gemini accepts 1M; the old ceiling was silently truncating our larger diffs, which would have made reviews shallow even once the auth was fixed.github_action_config.auto_describe: "false"— see the question below.contents: write→read. Review and suggest need neither, and write is a bad thing to hand apull_request_targetworkflow.v0.42.0instead of@main. Renovate'sgithub-actionsmanager (we runconfig:recommended) will keep it current, so the pin will not rot.Breaking change
Additional information
What I verified, and what I could not. Every config key and both model IDs were checked against PR-Agent's own source at the pinned tag (
configuration.toml,.secrets_template.toml,github_action_runner.py), the YAML parses, and theGEMINI_API_KEYorg secret does exist and is visible to this repo.I could not prove it end to end, and it is worth knowing why:
pull_request_targetandissue_commentalways execute the copy of the workflow living on the base/default branch, never the copy on a PR branch. So this file cannot run until it is merged.Concretely, the reviewer will not run on this PR at all, in either version — the old
pull_requesttrigger is gone from this branch's copy, anddevelop's copy has nopull_request_targettrigger yet. Confirmed: no PR-Agent job appears in this PR's checks, so please do not read its absence as a new failure. The first PR opened againstdevelopafter merge is the real test. I also have no way to check whether the Gemini key itself is valid and has quota.Three things worth a maintainer's call:
auto_describeoff. Previously it overwrote the PR description — the logs show it detectingExisting description was not generated by the pr-agentand rewriting it anyway. Since we write careful PR bodies by hand, I defaulted to protecting them. Flip one word if you would rather have it./reviewand/askneed this file onmaster, which currently has nopr_agent.ymlat all. Merging here fixes automatic review; the comment commands only come alive at the nextmastersync.issue_commenthas no author guard, so any GitHub user can comment/reviewon a PR and spend org Gemini quota. Restricting byauthor_associationis an option, at the cost of external contributors not being able to invoke it on their own PRs.One caveat on the pin: it locks the action definition only.
Dockerfile.github_action_dockerhubis a one-liner pullingpragent/pr-agent:github_action, a floating tag, so the agent itself is still whatever that image currently holds.🤖 Generated with Claude Code