# Retrieval evaluation kit

Publisher: Gigabrain Labs · Prepared 2026-09-05

This kit separates questions, relevance labels, retrieval results and answer review. The included lists are synthetic teaching data, not a Gigabrain benchmark.

## Download the runnable example

Save these files in the same directory:

- [Synthetic fixture](https://gigabrain.download/assets/resources/retrieval-scoring-fixture.json)
- [Python scoring script](https://gigabrain.download/assets/resources/score-retrieval.py)
- [Calculated result](https://gigabrain.download/assets/resources/retrieval-scoring-result.json)

Run with Python 3.11 or later: `python3 score-retrieval.py retrieval-scoring-fixture.json`

The script uses only the standard library, reads the named JSON file and prints JSON. It does not contact a model, search engine or network. The distributed lists were authored to demonstrate arithmetic. Execution verifies the calculations, not retrieval quality.

## Expected calculations

- Relevant returned: 4. Total returned: 12. Micro precision: 4/12.
- Relevant labeled: 6. Micro recall: 4/6.
- Per-query recalls: 1, 1, 0, 0.5. Macro recall: 0.625.
- Q3 retrieves no relevant passage. The aggregate must not hide it.

## Your query record

- Query ID, exact text and task type:
- Approved corpus and source revisions:
- Labeled relevant passage IDs and reasoning:
- Reviewer disagreement or incomplete labels:
- Retrieval configuration and result limit:
- Returned passage IDs and text:
- Run date and agent/tool versions:
- Supported answer, source scope and freshness:
- Failure and next investigation:

## Difficult cases

Include exact identifiers, paraphrases, changed facts, similar projects, ambiguous names and absent evidence. For absent evidence, evaluate whether the answer acknowledges the gap. Do not insert a zero-denominator recall into the average. The script reports those rows separately.

A duplicate passage ID within one list is rejected to prevent accidentally inflating counts. Empty returned lists have undefined precision (`null`) and zero recall when relevant evidence exists. The script does not grade answer correctness or source authority.

## Compare fairly

Keep corpus, questions and relevance criteria fixed. Separate tuning queries from held-back questions. Report individual important cases and label uncertainty alongside aggregates. Rerun material cases after an agent, source type or retrieval configuration changes.