THE SHORT ANSWER
Evaluate RAG retrieval separately from the eventual answer so you can distinguish missing knowledge from knowledge that was not found. Connect representative questions to expected evidence and inspect ingestion, filters, ranking and delivered context. Only then assess whether the model used the retrieved information correctly. This separation makes remediation more focused and prevents every weak answer from triggering another prompt revision.
To connect individual AI applications with reliable marketing operations, read the pillar guide AI transformation in B2B marketing: from individual prompts to reliable operations.
Establish expected evidence before searching
Take authorised real questions and ask specialists to mark the passages needed in the source collection. Record more than a keyword: specify product, market, timing and any exception required for a valid answer. Some questions need several sections. Also identify cases with no reliable answer in the collection. These test handling of knowledge gaps. Without this reference, attractive result lists cannot establish whether retrieval supplied the professionally necessary foundation or simply found documents with related wording. Retain the reasoning for each expected passage so later reviewers can understand why it belongs in the case. Explore this further in the guide A RAG source register: knowledge with owners and expiry dates.
Distinguish the stages where failure occurs
First check whether the source exists, is authorised and was ingested correctly. Then inspect whether filters permit it, retrieval finds it as a candidate and any reranking leaves it in the delivered context. Assess the answer afterwards. A correct source can disappear along the path even when the index generally works. A technically similar match can be wrong for the requested market. Preserve actual intermediate results and their versions. Diagnosis should identify the stage where required evidence disappears rather than calling every unsuccessful case a hallucination and directing all remediation toward the answer model. Explore this further in the guide Preparing RAG documents: meaningful sections rather than fragments.
Measure coverage and distracting results separately
Ask whether required evidence is present in the context the answer stage can actually use and how much irrelevant material accompanies it. A hit rate alone cannot reveal missing conditions. When several passages are necessary, check that they are available together. Define which search positions reach the answer context; a metric covering lower-ranked results may otherwise report success using information the model never receives. Break down results by question type. Exact product identifiers, free-form problem descriptions and multi-part requests can expose different weaknesses that one overall average quietly hides from the people deciding what to improve.
Compare changes under controlled conditions
Keep questions, source versions and assessment rules constant while changing segmentation, search methods, filters or ranking. Isolate changes where practical so improvements can be attributed. Anthropic describes contextual enrichment and combined retrieval approaches as design options to evaluate, not substitutes for measuring your own system. Do not reuse published results as a forecast for your collection. Track effort and runtime too. A method may retrieve more relevant evidence yet be too slow for the intended workflow. Decisions therefore need professional coverage and operational suitability together, with any trade-off stated clearly rather than hidden behind one favourable search metric.
Diagnose an explicitly fictional retrieval failure
A fictional knowledge agent must explain integration availability for a specific market. The valid regional list exists but an incorrect product filter excludes it. The model receives an older overview and gives an unsuitable answer. In this hypothetical test the defect belongs to filtering, not answer-prompt wording. A second variation supplies the right list but the model ignores a footnote. That variation tests interpretation. Keeping these cases separate prevents one corrective action from being presented as a solution to two distinct causes and shows which team needs to investigate each issue.
Connect findings to specific maintenance work
Each failure class should lead to an appropriate action: missing knowledge to the source owner, damaged extraction to document processing, unsuitable filtering to integration and incorrect use to answer logic. After correction, test the original case and related questions. Record remaining gaps and the acceptable response while they persist. The agent may ask for clarification or limit its answer when evidence is inadequate. An honest knowledge boundary is not a retrieval defect when the necessary information genuinely does not exist. Preserve this distinction in operational reports and release decisions so teams do not optimise search to compensate for absent foundations.
INTERACTIVE / FROM READING TO ACTION
Three checkpoints for your implementation
Choose a checkpoint and track your reading progress. Checks are neither saved nor sent and do not score your business.
CHECKPOINT 01
Measure coverage and distracting results separately
Ask whether required evidence is present in the context the answer stage can actually use and how much irrelevant material accompanies it.
Explore in the article ↗CHECKPOINT 02
Compare changes under controlled conditions
Keep questions, source versions and assessment rules constant while changing segmentation, search methods, filters or ranking.
Explore in the article ↗CHECKPOINT 03
Diagnose an explicitly fictional retrieval failure
A fictional knowledge agent must explain integration availability for a specific market.
Explore in the article ↗Frequently asked questions
Is assessing the final answer enough?
No. Without separate retrieval assessment you cannot tell whether evidence was absent or misused.
Does every retrieved section count as relevant?
No. Check scope, timing and the required claim rather than similar wording alone.
What should we do about missing knowledge?
Assign maintenance to the accountable source owner and define acceptable handling of the gap rather than only tuning retrieval.
How can I put “Establish expected evidence before searching” into practice?
Take authorised real questions and ask specialists to mark the passages needed in the source collection. Record more than a keyword: specify product, market, timing and any exception required for a valid answer. Some questions need several sections.
What matters when working on “Distinguish the stages where failure occurs”?
First check whether the source exists, is authorised and was ingested correctly. Then inspect whether filters permit it, retrieval finds it as a candidate and any reranking leaves it in the delivered context. Assess the answer afterwards.
