Enhanciar answers questions by retrieving pages from your company wiki and handing them to a model. If retrieval is wrong, everything downstream is wrong. This page publishes what we actually measured, how we measured it, and — at least as importantly — what these numbers do not tell you.
f051383
60 golden questions against the 40-page fixture wiki, scored per question type.
Baseline is the retrieval path as it shipped before this work
(all graph-memory flags off). Routing on is the same corpus and the
same questions with query_routing and relation_triples
enabled.
| recall@5 | MRR | citation precision | |||||
|---|---|---|---|---|---|---|---|
| question type | n | baseline | routing on | baseline | routing on | baseline | routing on |
| lookup | 15 | 1.00 | 1.00 | 0.90 | 0.90 | 0.21 | 0.32 |
| conceptual | 15 | 0.93 | 0.93 | 0.79 | 0.82 | 0.21 | 0.30 |
| relational | 15 | 1.00 | 1.00 | 0.86 | 0.86 | 0.23 | 0.34 |
| impact | 8 | 0.83 | 0.83 | 0.65 | 0.68 | 0.25 | 0.25 |
| code | 7 | 1.00 | 1.00 | 1.00 | 1.00 | 0.62 | 0.71 |
| overall | 60 | 0.96 | 0.96 | 0.84 | 0.85 | 0.27 | 0.36 |
Rounded to two decimals for display. The full-precision runs are committed at
tests/eval/baseline.json (recorded at commit dd8e8a2, and it
reproduces byte-identically at f051383) and tests/eval/routed.json
(commit f051383). A test in the repo compares this table against those two
files and fails if they disagree.
impact keeps all 8 because a blast
radius genuinely is spread out). The win is citing fewer pages, not finding
more of them.conceptual +0.03 and
impact +0.02 — the only two buckets where it showed up at all. That is
the entire measured contribution of typed-triple traversal to ranking. It is small,
and we are not going to describe a +0.01 overall as a breakthrough.The fixture wiki is 40 pages and retrieval returns up to 8 of them. At that corpus size almost any relevant page lands in the result set no matter what the ranker does, so recall@5 has very little room to move in either direction. A 0.96 here is not evidence of good retrieval at scale — it is evidence that the haystack is small. We publish it because a drop would be meaningful, not because the level is.
The harness deliberately runs offline: vector_search is off, so no
embedding API is called and the eval needs no key and no network. But that means
retrieval produces a single ranked list, and Reciprocal Rank Fusion preserves the order
of a single list at any positive weight. The keyword/vector re-weighting half of the
routing table — lookup keyword-weighted, conceptual
vector-weighted — therefore has no effect in this eval and is untested by these
numbers. It only does anything in a workspace with vector search enabled.
Measuring it needs a vectors-on eval mode, which does not exist yet.
The usefulness prior learns from real usage — which offered pages an answer actually
cited. This corpus has never been used by anyone, so the prior is empty and its
contribution is precisely zero. We verified that rather than assuming it: re-running
with retrieval_feedback on produces the identical table, to the decimal.
That layer's value, if it has any, shows up after real traffic and is not
represented anywhere in these numbers.
We wrote the 40 fixture pages and we wrote the 60 questions, which is a real conflict of interest: a golden set authored alongside the system it grades will flatter it in ways we cannot fully see from the inside. Both are committed in full so the bias is at least inspectable. Relatedly, this is not a competitive benchmark — we have not run any other product on this corpus, we are not claiming a comparison, and the numbers should not be read as one.
lookup, conceptual
and relational have 15 questions each, impact has 8 and
code has 7. The small buckets are noisier — one question is worth 0.125
of the impact score.
tests/eval/corpus/ — 40 markdown pages laid out exactly like a real
workspace wiki, for one invented company: 12 docs/ (a PRD,
design docs, runbooks), 10 conversations/ (Slack-style
thread digests), 10 tickets/,
8 entities/ (services and people), plus a pre-built
_index/triples.json so the typed-relation traversal has something to walk.
The harness copies this tree into a temp directory and repoints both
WIKI_ROOT constants at it, so it can never touch real data.
tests/eval/golden.jsonl — 60 lines of
{question, expected_pages[], type}. expected_pages is the set
of pages a correct answer would have to be grounded in; type is one of the
five routing classes: lookup, conceptual,
relational, impact, code. A malformed line is a
hard error in the harness rather than a skipped question, because a silently dropped
question would quietly inflate every score.
Every question goes through the real production retrieval function,
enhanciar.server._gather_wiki_context — the same code path a live chat
question takes, including the ACL check. Nothing is stubbed or simulated. The returned
sources are reduced to page keys (category/name), which is the identical
notion of "the same page" the fusion code uses, so a metric can never disagree with the
thing it is measuring.
No API key, no network and no production data — the run takes about seven seconds on a laptop.
git clone https://github.com/enhanciar/enhanciar.git
cd enhanciar
git checkout f051383
pip install -r enhanciar/requirements.txt
# baseline column (all graph-memory flags off)
python scripts/run_retrieval_eval.py
# "routing on" column
ENHANCIAR_FF_QUERY_ROUTING=1 ENHANCIAR_FF_RELATION_TRIPLES=1 \
python scripts/run_retrieval_eval.py
# markdown table, e.g. for a PR comment
python scripts/run_retrieval_eval.py --markdown
Script: scripts/run_retrieval_eval.py ·
corpus: tests/eval/corpus/ ·
questions: tests/eval/golden.jsonl ·
raw runs: tests/eval/baseline.json, tests/eval/routed.json.
The script always exits 0 by design — a retrieval regression should be visible on the
pull request, not a red X that trains people to re-run CI until it passes.
Ingest enrichment extracts subject → predicate → object facts from each document, each carrying the exact sentence that supports it, stored per workspace. On its own this moved retrieval by exactly nothing — measured, all six scores at +0.00 — because it only extracts and stores; nothing read the triples until routing shipped. Its measurable effect appears in the row below; its real product value is the evidence quote shown to the user, which this benchmark does not score.
Classifies each question into one of the five types and sets the fusion weights, whether
the triple graph is traversed, and the page budget. This is where the whole delta in the
table comes from: citation precision 0.27 → 0.36 (almost entirely the
page budget) and MRR 0.84 → 0.85 (traversal, on
conceptual and impact only). Worth recording: the traversal
weight we originally designed measurably hurt relational and impact MRR during
development — a single triple could outrank the best keyword hit outright — so triples
were retuned down to a tie-breaker before shipping. The plan was wrong and the eval is
what caught it. (That rejected configuration no longer exists in the code, so unlike
everything else on this page its numbers are not reproducible from the repository; the
development record is in PLAN_GRAPH_MEMORY.md.)
Records which offered pages an answer cited and feeds a capped, time-decayed usefulness prior back into ranking. Contributes nothing to this table, for the structural reason in limitation 3: a corpus with no usage history has no signal to learn from. Included here only so the page accounts for every shipped change rather than quietly omitting the one with no number attached.