RAG evidence flow and failure diagnosis

Follow the same evidence through retrieval, ranking, context assembly, and the answer to see where the information needed for a correct answer can be lost.

Learning objectives: Use required evidence to locate the first failing stage, and compare the conditions and costs of different fixes.

Content checked: 2026-10-04 · Each level has independent explanations, tasks and inspections

Choose a starting point based on your familiarity with this topic. Current level: Implementation · Build it. After completing the task, continue to the next level. Reading and self-checks alone do not establish mastery.

Review the prerequisites

Suitable for: Able to read lists and collections to prepare diagnostic tools.

Learn first:Context and generation budgets →Learn first:Tool calls: structure, authorization, and business contracts →

recall: Finding potentially relevant candidates from the database does not mean that they will eventually enter the model input.
rearrange: Re-prioritize candidates, affecting which material is placed first in a limited context.
necessary evidence: A collection of materials sufficient to support the conclusion of this question may require both text and exceptions.
evidence support: Whether the conclusion is supported by actual materials and applicable conditions; there may or may not be support with reference links.

How does the mechanism work?

Confirm information
The original material exists, is valid, and has the right to be used.
Recall and sort
Record candidate identification and ranking.
Assembly context
Maintain identification of necessary materials, conditions and sources.
Generate and verify
Check whether each conclusion and rejection is consistent with the evidence.

Implementation · Build it

Track evidence through a fixed retrieval pipeline

Objectives of this level: Ability to run evidence experiments and locate context-clipped exception errors.

Fix the variable parts of the experiment

rag_evidence.py performs no search or generation. It fixes candidates, ranked, and context ID lists, and required contains both policy-current and policy-exception. This makes evidence movement reproducible without model randomness.

Locate the first missing stage

Candidates and ranking contain both required passages, but context keeps only the first. diagnose returns context, locating the first gap at assembly for this input. Keeping both passages returns supported. That checks a supplied evidence set; it neither generates nor reviews an answer.

Match metrics to the labels

A task needing one source can use that source's recall and rank. A conclusion requiring several conditions needs the whole required set checked. Production must also bind document versions, permissions, passage text, and citations. Matching identifier strings alone establishes neither completeness nor semantic support.

Run experiments and observe counterexamples

Pipeline diagnostics for fixed evidence identification lists; does not perform vector retrieval, ground truth reranking, language modeling, or semantic refereeing.

Python 3.10+ · Runs by default using only the standard library · Runs on your computer

Check the two pieces of necessary evidence
Running context missing counterexample
Change candidate and answer support conditions separately

Downloadrag_evidence.py ↓

python3 rag_evidence.py

View the entry-point script

"""Trace evidence survival through fixed lists; no search or LLM is executed."""
import json


def diagnose(required, candidates, ranked, context, supported_claims):
    if not required.issubset(set(candidates)):
        return "retrieval"
    if not required.issubset(set(ranked)):
        return "ranking"
    if not required.issubset(set(context)):
        return "context"
    if not supported_claims:
        return "generation"
    return "supported"


def demo():
    required = {"policy-current", "policy-exception"}
    candidates = ["old-policy", "policy-current", "policy-exception"]
    ranked = ["policy-current", "policy-exception", "old-policy"]
    stage = diagnose(required, candidates, ranked, ranked[:1], True)
    fixed = diagnose(required, candidates, ranked, ranked[:2], True)
    assert stage == "context" and fixed == "supported"
    return dict(first_failure=stage, fixed_evidence_check=fixed,
                generation_still_needs_review=True)


if __name__ == "__main__":
    print(json.dumps(demo(), sort_keys=True))

Expected output when running locally

{"first_failure": "context", "fixed_evidence_check": "supported", "generation_still_needs_review": true}

Locating the earliest evidence gaps
Collection checking separate from semantic checking
Final repair needs to verify the true conclusion

View the running environment, output and verification records →

Acceptance task for this level

Run rag_evidence.py, then remove the exceptions from candidates, and finally keep the evidence intact but change supported_claims to False.

Check each item after completion

Original experiment first_failure as context
Positioned as retrieval after deleting candidate evidence
When the evidence is complete but the conclusion is not supported, it is positioned as generation

Save your own processes, code and results. Acceptance requirements are provided here, and course mastery status will not be automatically graded or saved at this time.

Hide the answer and check your understanding

The experiment returns supported. Does it prove that the answer of the real RAG is correct?

Expand reference derivation

No. It only verifies the teaching identification set and the given Boolean condition; the real answer requires checking the text, conditions, quotes and conclusions one by one.

Further explanations and practice

When encountering unfamiliar principles, first read the implementation, continuous questioning and migration cases, and then independently explain the premise and boundaries. Answers and notes are saved to the original account record.

Evidence flow and stage-by-stage RAG diagnosisReading principles, implementation and continuous questioning →Intermediate · System design · Answer independently →

Semantic completeness and evidence location in chunkingReading principles, implementation and continuous questioning →Intermediate · Implementation · Answer independently →

All linked explanations and exercises (4 )

Sources and verification scope

The principles are based on public information; the numbers, cases and tasks are the teaching design of this website. Offline experiments verify the range noted on this page, and the learning effect still needs to be judged through independent tasks and feedback.