Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion docs/quality/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ CodeyBox ships these audit-type presets as YAML resources (see `docs/quality/pre
| `quality` | LLM review focus for dead code, magic numbers, naming, error handling. |
| `completeness` | LLM review focus for TODOs, missing tests, half-finished impls. |
| `cheating` | Deterministic diff-patterns + LLM review focus for agent shortcuts. |
| `tests` | Deterministic diff-patterns for no-op assertions + LLM review focus for test meaningfulness. |
| `tests` | Deterministic diff-patterns for no-op assertions + LLM review focus for test assertion quality (anti-gaming). Test existence/completeness is owned by the deterministic `tests:coverage` gate below. |

A project enables a preset by listing its name in
`Audit.AuditTypes` (see `docs/concepts/projects.md`).
Expand Down Expand Up @@ -467,6 +467,23 @@ tool auditor (`Required = None`), auto-included by the composer like
diff. A project opts out via `ExcludedAuditors`. See
[`coverage.md`](coverage.md) for the full configuration.

### Meaningfulness review vs coverage gate

Test existence and test quality are owned by different auditors by design, so the
audit converges instead of re-litigating the same gap on every iteration:

* `tests:coverage` (deterministic) owns **existence / completeness**: every executable
line changed in the item's diff must be exercised by a test. It is stateless per
iteration and never judges whether an assertion is any good.
* `tests:meaningfulness-review` (LLM) owns **assertion quality / anti-gaming only**,
bounded strictly to tests added or modified in the diff: tests that execute code but
assert nothing, tests that assert on a mock/stub instead of the result under test,
missing error/edge-path assertions inside those tests, and coverage-padding tests
written only to satisfy the gate. It must never enumerate untested internal methods,
demand tests for code outside the diff, or report the mere absence of a test — that
is the coverage gate's job. Like the gate, it is stateless / isolated per audit
iteration, with no cross-iteration memory.

## Rework prompt

When an audit iteration fails, `ReworkPromptBuilder` assembles a prompt
Expand Down
11 changes: 8 additions & 3 deletions src/CodeyBox.Audit.Llm/LlmReviewAuditor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,14 @@ public string? SelfReviewGuidance
""";

private const string TestsGuidance = """
- **Adequacy**: Ensure each new public class, function, endpoint, and error path has at least one test.
- **Meaningfulness**: Avoid implementation-mirroring, pure-mock tests, no-assertion tests, and trivially-true assertions.
- **Edge cases & failures**: Cover boundaries, empty, null, unicode, timeouts, network errors, and resource exhaustion.
- **Assertion quality, not existence**: test existence and line coverage belong to the
deterministic tests:coverage gate — never ask for a test that does not exist and never
enumerate untested methods or branches.
- **Gaming tests**: flag added/modified tests that execute code but assert nothing, assert
on a mock/stub instead of the SUT-produced value, or pad coverage without verifying
behavior.
- **Edge assertions within the test**: the added test should assert the error/edge paths
it executes, not just the happy path.
- **Heuristic**: Ask yourself: "if I introduced a plausible bug (off-by-one, inverted condition, forgotten null check), would this test catch it?"
""";

Expand Down
Loading
Loading