AW Dev Rethought

✂️ Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away - Antoine de Saint-Exupéry

🧠 AI with Python — 📚 Simple Retrieval-Augmented Generation (RAG)


Description:

A model does not automatically know the contents of your documents. RAG retrieves relevant passages and supplies them as context when asking the model a question.

This example uses four fictional store-policy paragraphs, Sentence Transformers, FAISS and a local Ollama model. The accompanying notebook runs the entire pipeline and saves actual replies.


1. Prepare the documents

Each short paragraph is one chunk. Keep its ID, filename and text together:

{"id": "S1", "source": "returns.md",
 "text": "Demo Store accepts returns within 30 days of delivery. "
         "Returned items must be unused and in their original packaging."}

The other passages cover refunds, shipping and support. Longer documents need a suitable chunking strategy; these tiny paragraphs already fit the embedding model’s input limit.


2. Embed and index

Install sentence-transformers, faiss-cpu and numpy. Separately, install Ollama and run ollama pull llama3.2:1b in a terminal. The notebook keeps installation and imports in separate cells.

vectors = encoder.encode(
    [c["text"] for c in chunks], normalize_embeddings=True
).astype("float32")
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)

The all-MiniLM-L6-v2 encoder produces 384-dimensional vectors. Normalize the query too, so inner product corresponds to cosine similarity. Model downloads need internet; this example generates answers locally afterward.


3. Retrieve relevant passages

question = "How long do I have to return an item?"
hits = retrieve(question, k=2)

Our executed run retrieved:

Rank Source Cosine similarity
1 S1 · returns.md 0.680
2 S2 · refunds.md 0.531

These are similarity scores, not answer-confidence percentages. Top-k search returns matches even when no passage answers the question.


4. Augment the prompt, then generate

context = "\n\n".join(f"[{h['id']}] {h['text']}" for h in hits)

Send the context and question to Ollama’s /api/chat endpoint. The system instruction asks the model to use supplied facts and abstain when the answer is absent. We request JSON fields answer and source_ids so the application can display them consistently. This constrains structure, not correctness.

The full retrieval and generation code is inside the notebook’s rag() function:

result = rag(question)
print(result["answer"])
print(result["sources"])

Recorded output:

30 days
{'S1': 'returns.md'}

This is a real local model response, not a hard-coded answer. RAG adds information to the prompt; it does not train the language model on these documents.


5. Test a missing answer

We also asked: Does Demo Store offer a lifetime warranty? The retrieved passages did not provide that information.

The model answered: “I do not have that information in these documents.” It nevertheless attached S1 and S4 in its raw JSON. The application suppresses citations for that exact abstention message; the notebook retains the unmodified model reply for inspection.

This small test demonstrates one observed abstention, not reliable detection of every unanswerable question.


What to keep in mind

  • Source-ID checks confirm that an ID was retrieved; they do not prove factual support.
  • Retrieved content can be outdated, irrelevant or contain malicious instructions. Prompt wording alone is not a security boundary.
  • Use access controls, evaluation and suitable document refresh/chunking before building a production system.
  • Outputs can vary across model/runtime versions, even with a fixed seed and temperature zero.

Retrieve → augment → generate. Keep the evidence available so the answer can be checked.


Link copied!

Comments

Add Your Comment

Comment Added!