Skip to content

CI: make the Gemini PR reviewer actually review - #1140

Merged
Gui-FernandesBR merged 1 commit into
developfrom
ci/pr-agent-gemini-model
Aug 12, 2026
Merged

CI: make the Gemini PR reviewer actually review#1140
Gui-FernandesBR merged 1 commit into
developfrom
ci/pr-agent-gemini-model

Conversation

@Gui-FernandesBR

@Gui-FernandesBR Gui-FernandesBR commented Aug 12, 2026

Copy link
Copy Markdown
Member

Pull request type

  • ReadMe, Docs and GitHub updates

Checklist

  • Tests — not applicable (workflow-only change)
  • Lint — not applicable (no Python touched)
  • CHANGELOG.md — no action needed; and per the file's own header, GitHub maintenance does not belong there

Current behavior

The Gemini PR reviewer has never posted a review on a single PR, and it never went red either, which is why it went unnoticed for so long. There are four independent problems.

1. The actual failure: no model was ever selected.

The workflow passed GOOGLE_AI_STUDIO.GEMINI_API_KEY but never set config.model, so PR-Agent kept its OpenAI defaults (gpt-5.6, falling back to gpt-5.6-terra). With no OpenAI key present, LiteLLM used a placeholder and every one of the three tools died:

Error during LLM inference: litellm.AuthenticationError: AuthenticationError:
OpenAIException - Incorrect API key provided: dummy_key.
Failed to generate prediction with any model of ['gpt-5.6', 'gpt-5.6-terra']

PR-Agent catches that exception rather than propagating it, so the job still reported success. Example: run 31350336071.

2. Fork PRs could never work. A pull_request run triggered from a fork gets no secrets, so the key would have been empty regardless — and those runs did not even start, sitting in action_required awaiting approval (22 of the last 60). This is the majority of our traffic: 23 of the 27 currently open PRs come from forks.

3. synchronize was pure waste. It is not in PR-Agent's pr_actions, and handle_push_trigger defaults to off, so every push logged Skipping action: synchronize after spending ~30 s building the container. 38 of the last 60 runs were that no-op.

4. Comment commands never fired once. issue_comment always runs the workflow from the default branch (master), and this file only exists on develop — so /review, /describe and /ask were dead on arrival. Confirmed empirically: all 74 recorded runs of this workflow are pull_request, zero are issue_comment.

New behavior

  • config.model / config.fallback_models set to Gemini (gemini/gemini-3.5-pro, falling back to gemini/gemini-3.6-flash). This is the fix for the dummy_key failure.
  • pull_requestpull_request_target, so fork PRs get the key and run without manual approval. This is safe here because the job never checks the pull request out: PR-Agent reads the diff over the GitHub API, and the only step is a container action, so no untrusted code is executed.
  • synchronize dropped from the trigger list. github_action_config.handle_push_trigger is the knob if we ever want a re-review on each push.
  • config.max_model_tokens raised off the 32k default. Gemini accepts 1M; the old ceiling was silently truncating our larger diffs, which would have made reviews shallow even once the auth was fixed.
  • github_action_config.auto_describe: "false" — see the question below.
  • contents: writeread. Review and suggest need neither, and write is a bad thing to hand a pull_request_target workflow.
  • Action ref pinned to v0.42.0 instead of @main. Renovate's github-actions manager (we run config:recommended) will keep it current, so the pin will not rot.

Breaking change

  • No

Additional information

What I verified, and what I could not. Every config key and both model IDs were checked against PR-Agent's own source at the pinned tag (configuration.toml, .secrets_template.toml, github_action_runner.py), the YAML parses, and the GEMINI_API_KEY org secret does exist and is visible to this repo.

I could not prove it end to end, and it is worth knowing why: pull_request_target and issue_comment always execute the copy of the workflow living on the base/default branch, never the copy on a PR branch. So this file cannot run until it is merged.

Concretely, the reviewer will not run on this PR at all, in either version — the old pull_request trigger is gone from this branch's copy, and develop's copy has no pull_request_target trigger yet. Confirmed: no PR-Agent job appears in this PR's checks, so please do not read its absence as a new failure. The first PR opened against develop after merge is the real test. I also have no way to check whether the Gemini key itself is valid and has quota.

Three things worth a maintainer's call:

  1. I turned auto_describe off. Previously it overwrote the PR description — the logs show it detecting Existing description was not generated by the pr-agent and rewriting it anyway. Since we write careful PR bodies by hand, I defaulted to protecting them. Flip one word if you would rather have it.
  2. /review and /ask need this file on master, which currently has no pr_agent.yml at all. Merging here fixes automatic review; the comment commands only come alive at the next master sync.
  3. issue_comment has no author guard, so any GitHub user can comment /review on a PR and spend org Gemini quota. Restricting by author_association is an option, at the cost of external contributors not being able to invoke it on their own PRs.

One caveat on the pin: it locks the action definition only. Dockerfile.github_action_dockerhub is a one-liner pulling pragent/pr-agent:github_action, a floating tag, so the agent itself is still whatever that image currently holds.

🤖 Generated with Claude Code

The job has never posted a review. It passed the Gemini API key but never
set a model, so PR-Agent kept its OpenAI defaults (`gpt-5.6`, falling back
to `gpt-5.6-terra`) and every call failed with

    AuthenticationError - Incorrect API key provided: dummy_key

PR-Agent swallows that error, so the run still went green and the failure
was invisible. Setting `config.model` / `config.fallback_models` to Gemini
models is the actual fix.

Three other reasons it never worked:

- `pull_request` gives no secrets to fork PRs, and most contributions are
  fork PRs (23 of the 27 currently open). Those runs sat in `action_required`
  (22 of the last 60) and would have had an empty key even once approved.
  `pull_request_target` is safe here because the job never checks the PR out.
- `synchronize` is not in PR-Agent's `pr_actions`, so every push logged
  "Skipping action: synchronize" after a ~30 s image build - 38 of the last
  60 runs were that no-op.
- `issue_comment` runs the workflow from the default branch, and this file
  only exists on `develop`, so `/review` and `/ask` never fired once: all
  74 recorded runs were `pull_request`.

Also raise the token ceiling off the 32k default, since Gemini takes 1M and
larger diffs were being truncated; leave the human-written PR description
alone by turning `auto_describe` off; and drop the unused `contents: write`.

The action ref moves from `@main` to a release tag. Note that this pins the
action definition only: `Dockerfile.github_action_dockerhub` is a one-liner
that pulls `pragent/pr-agent:github_action`, a floating tag, so the agent
itself is still whatever that image currently holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Gui-FernandesBR
Gui-FernandesBR requested a review from a team as a code owner August 12, 2026 03:02
@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.80%. Comparing base (e0ff281) to head (c6c80a2).
⚠️ Report is 45 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1140      +/-   ##
===========================================
+ Coverage    82.18%   82.80%   +0.62%     
===========================================
  Files          122      129       +7     
  Lines        16355    16812     +457     
===========================================
+ Hits         13441    13922     +481     
+ Misses        2914     2890      -24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Gui-FernandesBR
Gui-FernandesBR merged commit 235487c into develop Aug 12, 2026
9 checks passed
@Gui-FernandesBR
Gui-FernandesBR deleted the ci/pr-agent-gemini-model branch August 12, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant