Skip to content

feat(cli): add jdf eval for RAG retrieval scoring (Recall@K, MRR, nDCG@K) - #6

Open
mmustafasenoglu wants to merge 1 commit into
uurtech:masterfrom
mmustafasenoglu:feat/eval-metrics
Open

mmustafasenoglu wants to merge 1 commit into
uurtech:masterfrom
mmustafasenoglu:feat/eval-metrics

Conversation

@mmustafasenoglu

Copy link
Copy Markdown

Summary

Starts on #3 (Benchmarking) by adding the missing measurement piece: a pure retrieval-metrics library plus a jdf eval command that scores a ranked run against relevance judgments (qrels) and writes a JSON report CI can diff over time for regression testing.

What's in here

  • tools/jdf-cli/src/lib/retrieval-metrics.ts — pure functions, no I/O: recallAtK, reciprocalRank (mean = MRR), ndcgAtK (exponential gains, log2 discount), plus evaluateRun which aggregates per-query means and skips queries with no judged-relevant docs.
  • tools/jdf-cli/src/commands/eval.tsjdf eval <qrels.json> <run.json> [--k N] [-o report.json]; prints a one-line summary and writes the full per-query report.
  • Tests (retrieval-metrics.test.ts, 15 cases on the built-in node:test runner, zero new dependencies) and sample fixtures under tools/jdf-cli/test/fixtures/.
  • test script wired into @uurtech/jdf-cli package.json; help text updated.
$ jdf eval eval/qrels.json eval/run.json --k 10
eval: 2 queries @10 — recall 1.000, MRR 0.750, nDCG 0.815
eval: report written to eval-report-k10.json

Verification

  • pnpm --filter @uurtech/jdf-cli test → 15/15 pass
  • pnpm --filter @uurtech/jdf-cli typecheck → clean
  • pnpm --filter @uurtech/jdf-cli build → success
  • End-to-end run on the checked-in fixtures produces the expected numbers (hand-verified).

Notes

  • CLI-only on purpose, following the existing chunk/embed precedent: this is RAG pipeline tooling, not a format change, so the three-surface rule does not apply.
  • Deliberately leaves the eval-set collection, chunker comparison, and CI gating from Benchmarking #3 as follow-ups; happy to take those on next if this direction looks right.

Related to #3

…G@K)

Adds a pure retrieval-metrics library plus a jdf eval command that
scores a ranked run against relevance judgments and writes a JSON
report CI can diff for regression testing. Covers the metrics,
automation, and regression-testing tasks of uurtech#3. Zero new
dependencies: tests run on node:test via the existing tsx dev tool.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant