feat(cli): add jdf eval for RAG retrieval scoring (Recall@K, MRR, nDCG@K) - #6
Open
mmustafasenoglu wants to merge 1 commit into
Open
mmustafasenoglu wants to merge 1 commit into
mmustafasenoglu wants to merge 1 commit into
Conversation
…G@K) Adds a pure retrieval-metrics library plus a jdf eval command that scores a ranked run against relevance judgments and writes a JSON report CI can diff for regression testing. Covers the metrics, automation, and regression-testing tasks of uurtech#3. Zero new dependencies: tests run on node:test via the existing tsx dev tool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Starts on #3 (Benchmarking) by adding the missing measurement piece: a pure retrieval-metrics library plus a
jdf evalcommand that scores a ranked run against relevance judgments (qrels) and writes a JSON report CI can diff over time for regression testing.What's in here
tools/jdf-cli/src/lib/retrieval-metrics.ts— pure functions, no I/O:recallAtK,reciprocalRank(mean = MRR),ndcgAtK(exponential gains, log2 discount), plusevaluateRunwhich aggregates per-query means and skips queries with no judged-relevant docs.tools/jdf-cli/src/commands/eval.ts—jdf eval <qrels.json> <run.json> [--k N] [-o report.json]; prints a one-line summary and writes the full per-query report.retrieval-metrics.test.ts, 15 cases on the built-innode:testrunner, zero new dependencies) and sample fixtures undertools/jdf-cli/test/fixtures/.testscript wired into@uurtech/jdf-clipackage.json; help text updated.Verification
pnpm --filter @uurtech/jdf-cli test→ 15/15 passpnpm --filter @uurtech/jdf-cli typecheck→ cleanpnpm --filter @uurtech/jdf-cli build→ successNotes
chunk/embedprecedent: this is RAG pipeline tooling, not a format change, so the three-surface rule does not apply.Related to #3