Retrieval benchmark

The numbers

60 golden questions against the 40-page fixture wiki, scored per question type. Baseline is the retrieval path as it shipped before this work (all graph-memory flags off). Routing on is the same corpus and the same questions with query_routing and relation_triples enabled.

Rounded to two decimals for display. The full-precision runs are committed at tests/eval/baseline.json (recorded at commit dd8e8a2, and it reproduces byte-identically at f051383) and tests/eval/routed.json (commit f051383). A test in the repo compares this table against those two files and fails if they disagree.

What actually moved

• Citation precision: 0.27 → 0.36 overall. This is the number that moved, and the honest explanation is unglamorous — retrieval used to return eight pages for every question regardless of how many the question needed, so most cited pages were noise. Routing sets a page budget per question class (lookup 4, conceptual/relational 5, code 3; impact keeps all 8 because a blast radius genuinely is spread out). The win is citing fewer pages, not finding more of them.
• MRR: 0.84 → 0.85 overall, from conceptual +0.03 and impact +0.02 — the only two buckets where it showed up at all. That is the entire measured contribution of typed-triple traversal to ranking. It is small, and we are not going to describe a +0.01 overall as a breakthrough.
• recall@5: 0.96 → 0.96, unchanged in every single bucket. That is the point of the column — it shows the precision gain was not bought by dropping relevant pages. See the limitations below for why recall@5 cannot do much more than that here.
• Nothing regressed. No bucket lost on any of the three metrics.

Limitations — what these numbers cannot tell you

1. recall@5 is near its ceiling. Treat it as a regression guard, not an achievement.

The fixture wiki is 40 pages and retrieval returns up to 8 of them. At that corpus size almost any relevant page lands in the result set no matter what the ranker does, so recall@5 has very little room to move in either direction. A 0.96 here is not evidence of good retrieval at scale — it is evidence that the haystack is small. We publish it because a drop would be meaningful, not because the level is.

2. Half the routing table is unmeasured, because the vector path never fires.

The harness deliberately runs offline: vector_search is off, so no embedding API is called and the eval needs no key and no network. But that means retrieval produces a single ranked list, and Reciprocal Rank Fusion preserves the order of a single list at any positive weight. The keyword/vector re-weighting half of the routing table — lookup keyword-weighted, conceptual vector-weighted — therefore has no effect in this eval and is untested by these numbers. It only does anything in a workspace with vector search enabled. Measuring it needs a vectors-on eval mode, which does not exist yet.

3. The self-improving retrieval layer contributes exactly nothing here.

The usefulness prior learns from real usage — which offered pages an answer actually cited. This corpus has never been used by anyone, so the prior is empty and its contribution is precisely zero. We verified that rather than assuming it: re-running with retrieval_feedback on produces the identical table, to the decimal. That layer's value, if it has any, shows up after real traffic and is not represented anywhere in these numbers.

4. Synthetic corpus, self-authored questions, single harness.

We wrote the 40 fixture pages and we wrote the 60 questions, which is a real conflict of interest: a golden set authored alongside the system it grades will flatter it in ways we cannot fully see from the inside. Both are committed in full so the bias is at least inspectable. Relatedly, this is not a competitive benchmark — we have not run any other product on this corpus, we are not claiming a comparison, and the numbers should not be read as one.

Also worth knowing

• Question types are not equally sampled: lookup, conceptual and relational have 15 questions each, impact has 8 and code has 7. The small buckets are noisier — one question is worth 0.125 of the impact score.
• The eval scores retrieval (which pages come back), not answer quality. It never calls a generation model, so it says nothing about whether the final answer was correct, well-written, or hallucinated.
• Scores are the mean over questions, unweighted. There are no confidence intervals; with n=60 a difference of a couple of hundredths is not something we would defend.

Methodology

The corpus

tests/eval/corpus/ — 40 markdown pages laid out exactly like a real workspace wiki, for one invented company: 12 docs/ (a PRD, design docs, runbooks), 10 conversations/ (Slack-style thread digests), 10 tickets/, 8 entities/ (services and people), plus a pre-built _index/triples.json so the typed-relation traversal has something to walk. The harness copies this tree into a temp directory and repoints both WIKI_ROOT constants at it, so it can never touch real data.

The golden set

tests/eval/golden.jsonl — 60 lines of {question, expected_pages[], type}. expected_pages is the set of pages a correct answer would have to be grounded in; type is one of the five routing classes: lookup, conceptual, relational, impact, code. A malformed line is a hard error in the harness rather than a skipped question, because a silently dropped question would quietly inflate every score.

What is actually run

Every question goes through the real production retrieval function, enhanciar.server._gather_wiki_context — the same code path a live chat question takes, including the ACL check. Nothing is stubbed or simulated. The returned sources are reduced to page keys (category/name), which is the identical notion of "the same page" the fusion code uses, so a metric can never disagree with the thing it is measuring.

The metrics (verbatim from the script)

recall_at_k(returned, expected, k=5)

"Fraction of the expected pages that appear in the top-k results."

mrr(returned, expected)

"Reciprocal rank of the FIRST correct page (0.0 if none appear). Answers 'how far does the reader have to scroll', which is what actually matters when the results become an LLM's context window."

citation_precision(returned, expected)

"Share of returned pages that are actually relevant. Deliberately measured over the WHOLE returned list, not the top k: every returned page is cited to the user and burns context budget, so an irrelevant page at position 8 is still a wrong citation."

Reproduce it

No API key, no network and no production data — the run takes about seven seconds on a laptop.

# from a checkout of the Enhanciar repository (the repo is private for now:
# email enhanciar@gmail.com and we will send the script, corpus and golden set)
git checkout f051383
pip install -r enhanciar/requirements.txt

# baseline column (all graph-memory flags off)
python scripts/run_retrieval_eval.py

# "routing on" column
ENHANCIAR_FF_QUERY_ROUTING=1 ENHANCIAR_FF_RELATION_TRIPLES=1 \
  python scripts/run_retrieval_eval.py

# markdown table, e.g. for a PR comment
python scripts/run_retrieval_eval.py --markdown

Script: scripts/run_retrieval_eval.py · corpus: tests/eval/corpus/ · questions: tests/eval/golden.jsonl · raw runs: tests/eval/baseline.json, tests/eval/routed.json. The script always exits 0 by design — a retrieval regression should be visible on the pull request, not a red X that trains people to re-run CI until it passes.

What each change contributed

Typed relation triples

Ingest enrichment extracts subject → predicate → object facts from each document, each carrying the exact sentence that supports it, stored per workspace. On its own this moved retrieval by exactly nothing — measured, all six scores at +0.00 — because it only extracts and stores; nothing read the triples until routing shipped. Its measurable effect appears in the row below; its real product value is the evidence quote shown to the user, which this benchmark does not score.

Query auto-routing

Classifies each question into one of the five types and sets the fusion weights, whether the triple graph is traversed, and the page budget. This is where the whole delta in the table comes from: citation precision 0.27 → 0.36 (almost entirely the page budget) and MRR 0.84 → 0.85 (traversal, on conceptual and impact only). Worth recording: the traversal weight we originally designed measurably hurt relational and impact MRR during development — a single triple could outrank the best keyword hit outright — so triples were retuned down to a tie-breaker before shipping. The plan was wrong and the eval is what caught it. (That rejected configuration no longer exists in the code, so unlike everything else on this page its numbers are not reproducible from the repository; the development record is in PLAN_GRAPH_MEMORY.md.)

Self-improving retrieval

Records which offered pages an answer cited and feeds a capped, time-decayed usefulness prior back into ranking. Contributes nothing to this table, for the structural reason in limitation 3: a corpus with no usage history has no signal to learn from. Included here only so the page accounts for every shipped change rather than quietly omitting the one with no number attached.

Numbers measured at commit f051383 on 2026-07-20 against tests/eval/corpus/ (40 pages) and tests/eval/golden.jsonl (60 questions), keyword-only path. · eval script on request · user guide · API reference