The short answer
Evaluate retrieved evidence and final answers separately, and keep difficult cases visible.
Build a query set, label relevant evidence, calculate precision and recall, and separate retrieval quality from the correctness of the final answer.Define what success means before looking at the answer
Start with the passages that should support the answer. Check whether retrieval found them, then whether the agent used them correctly. You need both checks: a convincing paragraph can hide a missed or misread source.
Start with questions you can judge yourself. Include straightforward lookups, paraphrases, changed facts and missing evidence. Preserve the corpus version so a later run has the same inputs.
Use a small labeled example
The downloadable example contains four fictional queries with predefined relevant passage IDs and returned lists. These lists were authored to explain scoring. They were not produced by Gigabrain or another search engine. The accompanying script calculates the results.
| Query | Relevant IDs | Returned IDs | Relevant returned |
|---|---|---|---|
| Q1 | A | A, X, Y | 1 |
| Q2 | B, C | B, C, Z | 2 |
| Q3 | D | X, Y, Z | 0 |
| Q4 | E, F | E, U, V | 1 |
Across these lists, four of twelve returned passages are relevant: micro precision is 4/12. Four of six relevant passages were returned: micro recall is 4/6. Per-query recalls are 1, 1, 0 and 0.5, so their mean is 0.625. These are different summaries of the same example.
Do not let one average hide the failure
Q3 fails entirely even though the aggregate recall looks better. Report each important query and useful groups alongside averages. A missing critical decision may matter more than several successful low-stakes lookups.
Precision measures the relevant share of retrieved material. Recall measures how much of the labeled relevant material was recovered. Both depend on the relevance labels. If you missed a valid source while labeling, your evaluation can mark a useful result as wrong.
Score missing evidence separately
A question with no supporting source has no relevant passage to retrieve. Do not hide it inside a recall calculation with a zero denominator. Test whether the final response acknowledges the gap and avoids unsupported details.
Likewise, a source can be retrieved without supporting the final claim. Add answer-level checks for source support, scope, freshness and refusal. Record disagreements between reviewers rather than forcing certainty into a single score.
Hold back some questions for review
Separate the questions used while tuning from a smaller set held back for review. Include a paraphrase you did not optimize against and a similar project that should remain separate. Keep changes to labels and sources in the record.
For a small test set, report how many questions passed and where the failures occurred. That gives readers more to work with than a broad reliability percentage. Repeat important queries in fresh sessions and retain failures. A new agent version, retrieval configuration or source type can justify a new run.
Use the kit to evaluate fit
Download the method, synthetic fixture and calculated results. Adapt the questions to a small project before importing your entire archive.
Gigabrain's public evidence covers specific release checks, not this synthetic ranking fixture. Compare packages only after checking your agent and deciding which acceptance cases matter to your work.
Compare the teaching fixture with an executed run
The scoring example above contains authored result lists. We also executed a separate four-mode retrieval experiment against the Full 1.1.0 archive.
Each of two fictional corpora contains seven queries. Six queries have labelled supporting sources. One has no supporting source. Four retrieval configurations produced 56 recorded runs in total.
The experiment calculates source recall from the labelled sources that appear in the result. It calculates source precision from relevant unique sources divided by returned unique sources. Duplicate hits do not create independent sources.
Open the per-query outputs, summary and runtime record.
Do not mistake verified citations for answer support
Every tested mode returned passages for the unsupported question. Matching source receipts therefore did not establish that a supported answer existed.
The development corpus's expansion mode recovered all labelled sources, while its mean source precision was approximately 28.6%. Keep both measurements visible. Neither number measures the quality of a generated answer.
The split named held-out was inspected during earlier development. This rerun treats it as regression data. Use new questions that your tuning process has not seen for an independent evaluation.
Download the runner instructions before extending the fixture.
Method note: Gigabrain Labs prepared this material with AI assistance. Fictional teaching examples and executed results are labelled separately.
