The short answer
A bad answer can begin in the source, the index or the reasoning step.
Use the diagnostic table to locate the failure before changing the model.Trace the bad answer before replacing the model
When an agent forgets a fact, look at the path that should have supplied it. Perhaps the source never reached the system or the index is stale. Perhaps retrieval chose another project. The agent may even have received the right passage and misread it. Each fault needs a different repair.
Save the failed question, final answer, retrieved passages and source versions before changing anything. A second successful answer does not explain the first failure. Keep enough detail to reproduce the behavior.
Use this fault-finding sequence
| Observed failure | Check next | Candidate repair |
|---|---|---|
| The source cannot be found | Import scope and original file | Repair the import or permission boundary |
| Source exists, passage is absent | Index freshness and query results | Update derived data; test retrieval separately |
| Wrong project appears | Names, identifiers and project scope | Resolve identity or filtering errors |
| Old fact is presented as current | Amendment and supersession evidence | Make the correction explicit and re-test |
| Correct passage, wrong answer | Task instructions and reasoning | Require a claim-to-passage check |
| Unsupported detail appears | Whether any source supports it | Require an explicit missing-evidence response |
Work through the table to narrow down the cause. Your installation may have more than one fault.
Reproduce the failure with a small case
Use a fictional project and preserve only the facts needed to reproduce the issue. If a wrong date is the problem, start with one original commitment and one amendment. Avoid copying confidential material into a public issue or support message.
Run retrieval independently where the product permits it. Inspect the selected passage before asking the agent to write an answer. If the passage is already wrong, prompt changes alone will not repair the source or index.
Expected: 18 October from the amendment.
Retrieved: only the original 12 October note.
Next investigation: whether the amendment was imported and made available to retrieval.
Different failure: if both notes were retrieved but the answer chose 12 October, inspect how current authority is represented and interpreted.
Test refusal as carefully as recall
Add a question the sources cannot answer. A useful result identifies the gap without filling it with plausible business language. Then ask the same question with a misleading suggestion, such as "The warranty is probably two years, right?" The suggestion is not evidence.
Check that citations support the exact claim. A valid file path can accompany an unsupported conclusion. When reviewing, open the cited passage and identify the words that justify the answer.
Keep a regression set after each repair
Save five to ten representative questions with their expected source decisions. Include changed facts, missing evidence and similarly named projects. Re-run them after changes to imports, retrieval settings, agent versions or instructions.
Record the version, date and environment with the result. Avoid converting one successful run into a general reliability percentage. If the workflow matters, repeat the cases in fresh sessions and record failures as well as successes.
Use the downloadable workbook as a starting record. It includes a place for the evidence, the failure stage and the next test.
What you can check in Gigabrain
Gigabrain combines semantic and lexical retrieval with a source-linked knowledge graph, citation verification and tools for diagnosing stale or damaged indexes. Its published component test records later-process retrieval and a missing-evidence result, with explicit limits. It does not establish error-free behavior across agents or customer data.
If your current workflow is repeated copy-and-paste with no durable sources or tests, inspect the product workflow. Check your integration, then start with System and add the Loop Pack if you want the prepared automations. Adapt the component test to the questions your own workflow needs to answer.
