FinRAG Eval

FinRAG Eval detects unsupported numerical claims in financial RAG.

Why this exists

The core question it answers: did the model assert a financial number that the retrieved evidence does not support?

Financial RAG can sound correct while getting the number wrong. A system can answer fluently, cite a plausible figure, and still be asserting a number that was never actually in what it retrieved. Standard RAG eval metrics (contextual precision/recall, answer relevancy) don't catch this — they check whether the answer resembles a reference answer, not whether a specific dollar or percent figure is traceable to specific retrieved text.

This matters because numeric grounding is not the same thing as answer correctness. A figure can be present in the retrieved evidence while the rest of the answer is still wrong (bad reasoning over a correct number), and a figure can be correct while being completely absent from what was retrieved (the model recalled it from training data, not from evidence — the dangerous case, because it looks identical to a properly grounded answer). FinRAG Eval checks grounding specifically, as a separate question from correctness, and never conflates the two.

Quick example

python -m src.eval.score_answer \
  --question "What was Apple's R&D spend?" \
  --answer "Apple's R&D expense was \$14.2 billion in fiscal 2024." \
  --context "Research and development expense was approximately \$31.4 billion."

{
  "verdict": "unsupported",
  "unsupported_confident": true,
  ...
}

No Docker, Ollama, database, or network call — just Python. The asserted $14.2B never appears in the retrieved context, so it's flagged, even though it reads as a fluent, confident answer.

What it detects

Four verdicts, computed deterministically (regex + unit/scale normalization, not an LLM judge, so the result is reproducible and free):

Verdict	Meaning
supported	Every financial figure the answer asserts is grounded in the retrieved context.
unsupported	At least one asserted figure is NOT grounded in the retrieved context — the dangerous case.
refused	The answer declined to state a figure rather than guess.
no_figure	The answer made no numeric claim at all (and wasn't a refusal).

Unit and scale normalization is handled so equivalent notations don't false-flag ($31.4B == $31.4 billion == 31,400 million), and accounting-negative notation is handled in both common forms ($(1,234) and ($1,234), (5.3%) and (5.3)%).

Real-world use cases

1. Financial RAG QA. A team building a financial document assistant can run FinRAG Eval before release to catch confident numerical claims that aren't backed by what was actually retrieved:

python -m src.eval.score_answer --question "..." --answer "..." --context "..."

2. Investment / equity research assistants. Applicable to RAG systems answering questions over 10-Ks, 10-Qs, earnings material, annual reports, and other financial disclosures — FinRAG Eval checks whether a generated financial figure is supported by the evidence that was actually supplied to the model, regardless of what document type it came from.

3. Financial-services internal AI. Teams building internal document-Q&A or research tools can use FinRAG Eval as an evaluation layer specifically for numerical grounding, alongside whatever other eval metrics they already run.

4. RAG regression testing. Teams changing embedding models, chunking, deduplication, reranking, routing, or retrieval parameters can run a fixed evaluation set before and after the change and compare numerical-grounding behavior directly. This repository's own Raw vs. Dedup vs. Routing experiment below is exactly this workflow, run for real.

5. CI / release quality checks. python -m src.eval.score_answer exits 2 on an unsupported verdict and 0 otherwise — this is real, current CLI behavior, verified, not a proposed feature — so a single answer can be gated directly in a shell pipeline. There is no packaged --fail-on-threshold flag for batch runs today; a batch gate means reading unsupported_confident_rate out of eval_runs/<run_id>.summary.json yourself and comparing it to a threshold you define.

6. Failure investigation. Engineers can inspect the full chain — question → generated answer → claimed financial figures → retrieved evidence → evaluator verdict — for any captured question, either directly in the JSONL row or in the dashboard's Failure Detail and Retrieval Evidence views (make ui).

7. Bring your own RAG stack. You do not need to adopt this repo's SEC/Ollama/pgvector pipeline to use the evaluator. The reference pipeline exists to give the evaluator something real to run against — the evaluator itself is the product:

pip install -e .

from finrag_eval import score_answer

result = score_answer(
    question=question,
    answer=model_answer,
    contexts=retrieved_evidence,   # list[str], one entry per retrieved chunk
)
result["verdict"]  # "supported" | "unsupported" | "refused" | "no_figure"

Zero Docker/Ollama/DB dependency for this path — verified by installing into a genuinely fresh virtual environment with no other project dependencies present.

Validated in Practice

This evaluator has been exercised against real financial documents and real RAG workflows, not just unit-tested in isolation. The full ticket-by-ticket evidence log with exact commands and output is docs/RELEASE_VALIDATION.md; this is the short version.

The standalone package installs and imports cleanly in a fresh, independent virtual environment with zero live-stack side effects. The reference pipeline has ingested real SEC filings (Apple FY2024 10-K, Apple FY2025 10-K, Microsoft FY2025 10-K — 156 chunks, confirmed via direct SQL query) and run live retrieval + local generation (Ollama) + deterministic scoring against them, producing both a genuine unsupported-figure case and a genuine honest-refusal case. Missing evidence is handled safely: zero retrieved chunks never reaches the generator — it's a fixed, deterministic refusal, not a chance for the model to guess from training memory.

The evaluator itself has been adversarially tested against unit/scale notation, accounting-negative formats in both common orderings, malformed input types, a stopped database, and other edge cases — several real bugs were found and fixed this way, each with a regression test. The three ablation configs (raw / dedup / metadata-routing) represent 90 independently captured, artifact-integrity-verified evaluation rows — no duplicate IDs, no partial rows, every summary statistic recomputed from the row-level data and matched exactly, not just trusted from a pre-aggregated file. The frontend has a clean npm ci install, passing lint/typecheck/build, and a live browser walkthrough of the failure-investigation path against real captured data. 114 offline tests pass on Python 3.11 and 3.12, both locally and in CI.

Not claimed: external customer adoption, enterprise deployment, production usage at scale, or use by any bank, hedge fund, or financial institution. This is an open-source evaluation tool, validated against real data by its own author — nothing more, nothing less.

Real results

Three complete, real, captured 30-question runs — not simulated, not partial:

Config	unsupported_confident_rate	pass_at_threshold	supported / unsupported / refused / no_figure
raw	0.7143	0.6667	4 / 10 / 15 / 1
+ dedup	0.6154	0.7333	5 / 8 / 17 / 0
+ routing	0.6667	0.7333	4 / 8 / 18 / 0

Read honestly: dedup and routing both lowered the unsupported-confident rate and raised pass@threshold relative to raw, but both also raised the refusal rate — some of the apparent grounding improvement is the system declining to answer more often, not only answering more accurately. Full breakdowns, methodology, and disclosed caveats: RESULTS.md.

DeepEval's 4 judge-model metrics (contextual precision/recall, faithfulness, answer relevancy) were skipped for these runs (--skip-deepeval) due to local memory constraints — disclosed, not silently omitted. unsupported_confident_rate and pass_at_threshold never depend on DeepEval either way. All three runs used OLLAMA_GEN_MODEL=llama3.2:3b rather than the documented default llama3 — also disclosed, not silent.

Initial manual finding (FY2024)

The first 3-question probe against Apple's FY2024 10-K is what originally exposed the core failure mode:

Question	Ground Truth	System Response	Verdict
Total net revenue vs FY2023?	$391.0B (+2% YoY)	Refused — "not found in context"	Honest refusal
Gross margin %?	46.2%	Refused — "not found in context"	Honest refusal
R&D spend + % of revenue?	$31.4B / 8.0%	Answered confidently with wrong figures	Confident hallucination

The dangerous failure mode isn't "I don't know" — it's "the answer is X" where X is wrong and sounds credible. Full manual transcript in notes.

How the evaluator works

src/eval/hallucination.py and src/eval/score_answer.py deliberately separate two questions that earlier logic in this project once conflated (a real bug, fixed, with regression tests proving it can't recur):

• Context grounding — is the figure the answer asserts supported by the retrieved context alone?
• Ground-truth correctness — does it match the expected answer?

The gold/expected answer can never create context grounding — a figure that happens to match the gold answer but wasn't actually retrieved is still unsupported. This is the invariant the whole project exists to protect.

score_answer() is the single reusable entrypoint: the CLI, the batch runner, and the frontend all consume its output, so none of them can disagree about what a verdict means.

Install

git clone https://github.com/Ruthwik-Data/finrag-eval
cd finrag-eval
python3 -m venv .venv && source .venv/bin/activate
which python   # should print .../finrag-eval/.venv/bin/python
make dev       # install dev/test deps
make test      # offline — no Docker/Ollama/DB/network required

make dev/make test/etc. use whatever python/pip is on your PATH — they don't create their own virtual environment, so activate one first (a very ordinary requirement, called out here because skipping it produces a confusing ModuleNotFoundError rather than a clear error).

Python API

pip install -e .   # editable install, not yet published to PyPI

from finrag_eval import score_answer

result = score_answer(
    question="What was Apple's R&D spend?",
    answer="Apple's R&D expense was $14.2 billion in fiscal 2024.",
    contexts=["Research and development expense was approximately $31.4 billion."],
)
result["verdict"]  # "unsupported"

contexts must be a list[str] — one entry per retrieved chunk, not one concatenated string; passing anything else raises a clear error rather than silently scoring garbage. See docs/PRODUCT.md for the full output contract.

CLI / batch evaluation

python -m src.eval.score_answer --question "..." --answer "..." --context "..."   # one answer, offline
make ask Q="What was Apple's FY2025 total net revenue?"                             # live retrieval + generation, requires the stack
make eval                                                                            # full 30-question batch, writes eval_runs/<run_id>.jsonl + .summary.json
make eval-dedup                                                                      # + chunk deduplication
make eval-routing                                                                    # + metadata-aware section routing
make ablation                                                                         # all three

Full walkthrough of every workflow with expected output: docs/USE_CASES.md.

Dashboard

make ui

A local dashboard (frontend/ — Vite + React + TypeScript) over the JSONL/summary artifacts: overview metrics with unsupported_confident_rate front and center, a run/ablation comparison, a filterable question explorer, a failure-detail view, and a retrieval-evidence view. Works immediately on bundled fixture data with no Ollama/Docker — every fixture screen carries a persistent "fixture" indicator, never mistaken for a captured result. To view a real make eval run instead:

node frontend/scripts/import-run.mjs eval_runs/<run_id>.jsonl

make ui-build runs the production build + type check.

Reference SEC workload

The reference pipeline this repo ships — real SEC EDGAR ingestion, local embeddings, pgvector retrieval, local generation — exists to give the evaluator something real to run against, not as the product itself:

SEC EDGAR (10-K, .htm)
    → parse_html_filing (Item-marker section splitting)
    → Chunker (configurable overlap, section boundaries preserved)
    → Ollama nomic-embed-text (local embeddings)
    → pgvector / Docker (local vector store)
    → Top-k retrieval
    → Ollama (local LLM, $0 API cost)
    → score_answer() — the same evaluator described above

make setup                              # starts Postgres/pgvector, prints required `ollama pull` commands
make download TICKER=AAPL COUNT=2       # Apple's two most recent 10-Ks
make ingest FILE=data/raw/<file> TICKER=AAPL
make ask Q="What was Apple's FY2025 total net revenue?"

Raw vs. dedup vs. routing experiment

Overlapping context chunks (10-20% overlap, standard for preserving table/section boundaries) turned out to be penalized by DeepEval's ContextualPrecisionMetric as independent retrieval failures — making eval scores worse as chunk quality improved. That's how this project's real numbers above (unsupported_confident_rate dropping with dedup) came to be measured in the first place: the ablation exists to make that kind of tradeoff visible and comparable instead of assumed.

Open-Source Impact

Real use of FinRAG Eval surfaced issues and prompted contributions beyond this repository. Verified directly against the GitHub API for this update — not copied from prior write-ups. Full audit trail, including what was checked and excluded, is in docs/CONTRIBUTIONS_AUDIT.md.

DeepEval

Building FinRAG Eval's ablation experiment (below) surfaced a real bug in DeepEval, the open-source LLM eval framework this project's --judge metrics are built on:

Problem. Real financial-document chunking uses 10-20% overlap to avoid cutting a table or footnote at a chunk boundary — standard practice for dense filings. DeepEval's ContextualPrecisionMetric scored each overlapping chunk as an independent retrieval, so increasing overlap (which improved retrieval and answer quality) made the metric worse. Redundancy was being scored as irrelevance.

Investigation. Reproduced with a concrete example: a revenue figure retrieved across 3 overlapping chunks (plus 2 narrative chunks), fully grounded, correctly answered — scoring 0.3-0.5 instead of near 1.0. Filed as confident-ai/deepeval#2594, with a minimal runnable repro.

Contribution. Opened PR #2743: groups RetrievedContextData chunks sharing the same source before scoring, and corrects the weighted-cumulative-precision formula to divide by the actual relevant-node count rather than total verdicts.

Result. Merged into DeepEval on 2026-06-14. src/eval/metrics.py's section_aware_precision/section_aware_recall in this repo are the local reference implementation of the same fix — unit-tested independently, against real Apple/Microsoft filings, not just DeepEval's own test suite.

Follow-on engagement from the same investigation, shown in full — not cherry-picked to the merged one:

#	Type	Status	Covers
#2743	PR	Merged (2026-06-14)	The fix described above
#2594	Issue	Closed by reporter, day after the fix merged	Original bug report + repro
#2775	Issue	Open	Feature request: per-document-type thresholds for heterogeneous filing types
#2788	Issue	Open	Parallel bug: ContextualRecallMetric has the same overlap-penalty issue
#2692	PR	Closed, not merged	Earlier attempt at regression fixtures for #2594, superseded by #2743
#2787	PR	Closed, not merged	Regression fixtures for #2743
#2789	PR	Closed, not merged	Regression fixtures for #2788
#2790	PR	Closed, not merged	Docs example for threshold_overrides

run-llama/llama_index

FinRAG Eval's --metadata-routing ablation (re-ranking retrieval by query-inferred document section — real, tested, in the captured results below) motivated two feature requests to llama_index proposing the same pattern as a first-class feature, each explicitly referencing this project:

• Issue #21862 — a metadata schema for section-aware routing in heterogeneous financial documents ("I've implemented a prototype of this approach in my finrag-eval project"). A follow-up PR, #22054 (StructuralRoleNodeParser), attempted an implementation two days later — closed, not merged.
• Issue #22032 — heterogeneous-index routing for mixed document types (10-Ks, earnings transcripts, balance sheets), citing this project directly.

Both issues are open and unresolved — filed, not yet acted on by maintainers.

Shubhamsaboo/awesome-llm-apps

Contributed PR #955, "Eval-First RAG" — a standalone Streamlit tutorial app porting FinRAG Eval's grounded/honest_refusal/confident_hallucination verdict framework as a teaching example for a widely-used community collection of LLM app examples. The PR description states directly: "Built on the eval-first approach behind my finrag-eval, where the same method surfaced a metric bug now merged into DeepEval." Closed, not merged. (The tutorial app itself uses an OpenAI key for its own minimal-setup design — that's the tutorial's choice, not FinRAG Eval's; this repo remains fully local/Ollama-only throughout.)

FinRAG Eval has therefore been used not only to evaluate its own reference financial RAG workload, but as a practical testbed that surfaced a real issue in upstream tooling and motivated related proposals elsewhere — one shipped, several still open.

Every link above resolves on GitHub and is independently verifiable — only #2743 shipped; the rest is disclosed as open or unmerged, not presented as additional wins.

Limitations

• 30-question eval is illustrative, not statistically significant.
• llama3-family models run locally are weaker than GPT-4-class models; all three captured runs used llama3.2:3b rather than the documented default llama3, disclosed above.
• DeepEval's 4 judge-model metrics haven't been captured for the 30-question dataset yet (memory-constrained development machine).
• Numeric-figure extraction is regex/heuristic, not an LLM judge — a number range written as one figure ("$10-12 billion") is still split into two; the refusal-vs-no_figure heuristic is pattern matching, not a classifier. Neither affects the headline unsupported_confident_rate. See RESULTS.md's Known Gaps for the full, current list.
• This is a numeric-presence checker, not an entity/date resolver — a figure that matches but comes from the wrong company or fiscal period reads as supported. Retrieval is expected to have already filtered by issuer/filing before evidence reaches the evaluator.
• Section detection (HTML and PDF parsers) is heuristic and may miss boundaries in non-standard filing formats.

Development & testing

make dev && make test

114 tests, offline, deterministic, no Docker/Ollama/DB/network required. See CONTRIBUTING.md for the PR checklist and the no-benchmark-claims-without-captured-evidence rule.

Security & data notes

SEC filings are public documents — nothing ingested here is sensitive. Offline scoring makes no network call. Live SEC ingestion talks only to sec.gov. Local generation/embeddings talk only to Ollama on localhost. No telemetry or analytics anywhere in the codebase. Never commit .env or real credentials (see .env.example); downloaded filing data (data/raw/, data/processed/) stays gitignored.

License

MIT — see LICENSE.