AW Dev Rethought

βœ‚οΈ Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away - Antoine de Saint-ExupΓ©ry

🧠 AI with Python – πŸ”— RAG Systems: Adding Source References 🐍


Description:

An answer says a refund is available within 30 days. A useful source reference lets the reader inspect the passage behind that claim.

In RAG systems, references should remain connected to the retrieved evidence throughout the pipeline. Adding a plausible-looking filename after generation does not establish that connection.


πŸ“š Keep Metadata with Every Chunk

Store the text alongside its source identity and location:

{
    "chunk_id": "support:f346ddf6:38-200",
    "title": "Customer Support Policy",
    "source": "support_policy.md",
    "section": "Refunds",
    "start": 38,
    "end": 200,
    "text": "Customers can request a refund within 30 days of purchase. ...",
}

This illustrative metadata comes from the notebook's fictional policy. The chunk ID contains a source-content hash and offsets, so unchanged content has a stable identity. Offsets use Python's [start, end) convention.

Use a real page number only if the document parser supplies one. Markdown sections and exact text offsets are sufficient for this example.


πŸ” Retrieve Evidence Without Losing Its Identity

Retrieve chunks using Sentence Transformers and FAISS, then assign response-local labels:

evidence = [
    {**hit, "citation_id": f"S{i}"}
    for i, hit in enumerate(hits, start=1)
]

S1 identifies a retrieved chunk within this response. It is distinct from the persistent chunk_id. Keep the label-to-source mapping in application code so filenames and locations come from known metadata.


🧾 Guide the Answer

The notebook builds a prompt with the question and labelled evidence. Its instructions ask the generator to:

  • Use only the supplied evidence.
  • Cite factual claims with the supplied IDs.
  • Avoid inventing IDs.
  • Say when the evidence is insufficient.

Treat retrieved passages as source data, not instructions. Prompting guides output but does not guarantee faithful citations.

To keep this lab local and reproducible, it does not call a generative model. It quotes a sentence from the top result and renders its reference:

Customers can request a refund within 30 days of purchase. [S1]

[S1] Customer Support Policy β€” Refunds, support_policy.md, characters 38:200.


βœ… Validate the Reference Chain

For an exact-quotation demo, validate that the ID exists and that the quote occurs in its cited evidence:

if source_id not in registry:
    raise ValueError("Unknown citation ID")

if quote not in registry[source_id]["text"]:
    raise ValueError("Quote not found in the cited evidence")

Then compare the retrieved chunk with its original file slice:

assert original[start:end] == chunk_text

The notebook verifies a valid reference, rejects invented ID S99, rejects an unsupported 365-day claim, and exercises the explicit no-evidence response. Only references used by the answer are displayed.


⚠️ Understand What These Checks Prove

An existing ID proves that a citation resolves. Exact quote matching proves that the quoted words occur in the selected passage. Neither proves that the source is true or that the passage answers the question.

Generated paraphrases require claim-level support checks beyond string matching. Evaluate whether factual claims are supported and whether citations cover those claims. Retrieval also needs an evaluated relevance and abstention policy for unanswerable questions; the notebook's empty-evidence test does not solve that automatically.


πŸ“Œ Key Takeaways

  • Attach source metadata before indexing and retain it during retrieval.
  • Bind response citation labels to known chunks.
  • Render references from the source registry.
  • Check traceability separately from factual support.
  • Preserve source versions and allow the system to say it lacks evidence.

A citation should make the evidence inspectable, not merely make an answer look credible.


Code Snippet:

None

Link copied!

Comments

Add Your Comment

Comment Added!