A RAG system that measures its own answers

The contract

Most RAG systems return a fluent answer and leave you to trust it. This one cannot return an answer without also returning the evidence and a measurement of how well that evidence supports it.

ragtrust index data/demo_corpus.md --out ./index
ragtrust ask "What is the purpose of max-pooling layers in CNNs?" --index ./index

Answer: Max-pooling reduces the spatial resolution of a feature map [2]. It also
enlarges the effective receptive field of later layers [1]. ...

Metrics:
  faithfulness           0.816
  contradiction_rate     0.727
  attribution            0.400
  attribution_precision  1.000
  attribution_recall     0.250
  relevance              0.419
  answer_relevance       n/a
  conciseness            0.679

Trust (arithmetic): 0.626
Trust (geometric):  0.597
Is trustworthy:     True

Ask something the corpus does not cover and it declines rather than confabulating. On the bundled corpus the pipeline answers 10 of 10 in-corpus questions and declines 10 of 10 out-of-corpus ones — mean top retrieval score 0.722 against 0.082.

Answering is deterministic. Generation defaults to greedy decoding with a fixed seed, so the same question against the same corpus returns the same answer and the same trust score. A measurement that moves when nothing it measures moved is not a measurement.

Five metrics, and a score that can't be gamed

Each measures something a different failure mode would break. Full definitions and proofs are in the formal spec.

Aggregation is non-compensatory

A weighted arithmetic mean lets fluency and relevance mask a hallucination. The geometric form collapses when any single dimension does, and by weighted AM–GM it can never overstate trust:

faithfulness=0.05, attribution=0.9, relevance=0.9, conciseness=0.95
  T_arith = 0.5700   <- passes
  T_geom  = 0.2863   <- correctly penalised

Both are reported. Weight choice is treated as a sensitivity question rather than an assumption: 10 000 Dirichlet weightings are sampled, and the two aggregators disagree on ranking in 1.90% of 450 000 comparisons.

A metric can also be undefined rather than zero — conciseness needs at least two claims, and answer relevance needs a backend that can back-generate questions. An undefined metric is dropped from aggregation, never scored as 0, and both aggregates renormalise over the surviving weights so the AM–GM bound keeps holding.

Every metric, measured on somebody else's data

Validating a metric only on data you wrote yourself measures the difficulty of your own test set, not the metric. Each of these was checked against independent human annotation — five benchmarks, none authored by this project.

Three results worth reading in full

Relevance is tested without circularity. The obvious approach — score the passages the retriever returned — is invalid, because the retriever uses the same embedder as the metric, so its top-k is selected by the quantity under test. Pairs are built from the judgments instead, with hard negatives chosen by BM25: lexical, and therefore independent of the embedder. Picking hard negatives with the dense retriever would bias the test the other way and is equally wrong.

Conciseness is strong where it is defined. Across all 4,143 summaries it scores 0.647 and cannot beat counting sentences — because 68.5% of real summaries decompose into fewer than two claims, where pairwise self-similarity has no meaning and the metric reports "undefined". Restricted to the 1,307 where it is genuinely computed it reaches 0.820 against 0.566 for the best pure-length baseline (p=0.0001). It also passes a discriminant check that could have gone the other way: SEAHORSE rates repetition and concise-representation separately, and the metric tracks repetition (0.647) while sitting at chance on concise-representation (0.514) — what "penalises padding, not length" requires, and not what a length proxy would do.

A shipped threshold was interrogated rather than assumed. support_threshold = 0.5 decides whether a citation counts as supported. Two benchmarks appear to disagree on its optimum (0.21 against 0.03) — but F1 ignores true negatives, so its optimum tracks class prevalence, and the two sets have opposite skew (33.5% vs 76.6% positive). Under a prevalence-independent criterion they agree closely: τ* = 0.20 and 0.29, both below 0.5. The default stays at 0.5 deliberately, because attribution feeds a trust score where a false "supported" inflates trust while a false "unsupported" only deflates it — the conservative error is the second one.

Each result, with confidence intervals, permutation tests and stated limitations, is in experiments/results/.

Retrieval, benchmarked

Dense, BM25, Reciprocal Rank Fusion and cross-encoder reranking are all implemented, and the default was chosen by measurement on BEIR/SciFact — 5,183 third-party documents, 300 third-party queries, third-party judgments.

All five non-baseline configurations beat dense at every k, each improvement surviving a paired bootstrap CI over 300 queries. Reranking wins by a further +0.043 and still defaults to off: it roughly doubles query latency and its unbounded scores are not comparable with the retrieval gate's threshold. That is a cost decision, documented at the config field, not a claim that it does not help.

The abstention gate is calibrated too — 300 answerable queries against 900 unanswerable ones drawn from three other BEIR sets, reaching ROC-AUC 0.962 pooled and 0.907 against the hardest same-domain tier.

Explore the real output

All 20 evaluation questions, scored against the passages actually retrieved. Precomputed — this page runs no models.