RAG Tutorial: Chat With Your College Notes PDF Using Python, Embeddings and Streamlit

In Part 2 you built StudyBuddy, a chat app that streams answers from a large language model inside Streamlit. It has one big blind spot: it knows nothing about your syllabus or the notes your professor shared last week. Ask about "the deadlock examples from Unit 3" and it will guess. In this part you fix that. StudyBuddy will read a PDF of your notes, answer from it with page citations, and say "I don't know" when the answer is not there.

The technique is RAG, retrieval augmented generation, one of the most widely used patterns in real world AI products and a strong fresher portfolio project.

What you will build

• rag.py: load a PDF, split it into overlapping chunks, embed them with a free local model, search with plain numpy
• prompt.py: a grounded prompt that cites numbered sources and admits when it does not know
• An upgraded Streamlit app with a PDF uploader, cached indexing and a Sources panel
• A small script that measures whether retrieval finds the right pages

What you need

• The StudyBuddy project from Part 2, with your .env holding LLM_BASE_URL, LLM_API_KEY and MODEL
• Any OpenAI compatible endpoint: a hosted free tier (Google Gemini, Groq and others offer one) or a local Ollama server
• A text based PDF of your notes (scanned image PDFs need OCR first)

pip install pypdf sentence-transformers numpy

Why fine tuning is usually not the first answer

The instinct is to fine tune, that is, train the model further on your documents. For "answer from my notes" it is usually the wrong first move:

1. Fine tuning teaches behaviour better than facts. It is good for a fixed format or tone, unreliable for recalling a specific line from page 14, and the model can still hallucinate.
2. Your notes change weekly. RAG indexes a new PDF in seconds; fine tuning means retraining.
3. No citations. A fine tuned model cannot tell you where an answer came from. RAG can, because you chose the chunks.
4. Cost and effort. Fine tuning needs prepared training data and GPU time or a paid service. RAG runs on a laptop with free tools.

Use RAG when the model needs to know something. Consider fine tuning only when it needs to behave differently and good prompting has already failed.

The RAG pipeline: load, chunk, embed, store, retrieve, augment, generate

The first four steps run once per document (indexing). The last three run on every question.

Step	What happens in StudyBuddy
Load	Extract text from each PDF page with pypdf
Chunk	Split text into overlapping pieces of about 800 characters
Embed	Turn each chunk into a vector of 384 numbers
Store	Keep vectors in a numpy matrix beside their text and page
Retrieve	Embed the question, find the top k most similar chunks
Augment	Paste those chunks into the prompt as numbered context
Generate	The LLM answers using only that context, with citations

An embedding is a list of numbers that captures meaning, so texts with similar meaning get similar vectors. "What causes deadlock?" and "the four Coffman conditions" share almost no words, yet a good embedding model places them close together. That is why RAG search beats Ctrl+F.

Here is the complete rag.py. The next sections explain each part.

import numpy as np
from pypdf import PdfReader
from sentence_transformers import SentenceTransformer

EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"


def load_pdf(file) -> list[dict]:
    """Return one dict per page: {"page": n, "text": "..."}."""
    reader = PdfReader(file)
    pages = []
    for i, page in enumerate(reader.pages, start=1):
        text = page.extract_text() or ""
        text = " ".join(text.split())  # collapse whitespace and line breaks
        if text:
            pages.append({"page": i, "text": text})
    return pages


def chunk_text(text: str, chunk_size: int = 800, overlap: int = 150) -> list[str]:
    """Split text into overlapping windows of roughly chunk_size characters."""
    if overlap >= chunk_size:
        raise ValueError("overlap must be smaller than chunk_size")
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + chunk_size, len(text))
        # try not to cut a word in half: back up to the last space
        if end < len(text):
            space = text.rfind(" ", start, end)
            if space > start + chunk_size // 2:
                end = space
        chunks.append(text[start:end].strip())
        if end >= len(text):
            break
        start = end - overlap
    return [c for c in chunks if c]


def chunk_pages(pages: list[dict], source: str, **kwargs) -> list[dict]:
    records = []
    for p in pages:
        for j, chunk in enumerate(chunk_text(p["text"], **kwargs)):
            records.append({"source": source, "page": p["page"], "chunk": j, "text": chunk})
    return records


class VectorStore:
    """A tiny in memory vector store: a numpy matrix plus a list of records."""

    def __init__(self, model: SentenceTransformer):
        self.model = model
        self.records: list[dict] = []
        self.vectors = None

    def add(self, records: list[dict]) -> None:
        texts = [r["text"] for r in records]
        vecs = self.model.encode(texts, normalize_embeddings=True, batch_size=32)
        vecs = np.asarray(vecs, dtype=np.float32)
        self.vectors = vecs if self.vectors is None else np.vstack([self.vectors, vecs])
        self.records.extend(records)

    def search(self, query: str, k: int = 4) -> list[dict]:
        if self.vectors is None or not self.records:
            return []
        q = self.model.encode([query], normalize_embeddings=True)[0]
        scores = self.vectors @ q  # cosine similarity, since all vectors are unit length
        top = np.argsort(-scores)[:k]
        return [{**self.records[i], "score": float(scores[i])} for i in top]

Loading and chunking with overlap

Extracting text with pypdf

PdfReader(...).pages gives you each page and extract_text() returns its text layer. Two details matter. We keep the page number with every piece of text, which is what makes citations possible later. And we collapse whitespace, because PDFs break lines mid sentence and those newlines waste chunk space.

Gotcha: if every page comes back empty, the PDF is a scan (images only). pypdf cannot read images; you would need an OCR step such as Tesseract first. The app shows a clear error for this case.

Why chunk, and why overlap

You cannot paste a 120 page PDF into every prompt: context windows are limited and long prompts are slower. Also, an embedding of a whole chapter is a blurry average of many topics. Small focused chunks give sharp embeddings that match specific questions.

Overlap solves the boundary problem. If a definition starts at the end of chunk 4 and finishes in chunk 5, neither chunk holds the whole idea. With 150 characters of overlap, the tail of each chunk repeats at the head of the next, so a sentence crossing a boundary appears whole in at least one chunk.

chunk_text slides a window across the text, backs up to the nearest space so words are not cut, then starts the next window overlap characters early. It also raises an error if overlap >= chunk_size, because the window would never advance and the loop would never end.

Sizes: start with 500 to 1,000 characters and 10 to 20 percent overlap, then tune with the evaluation script below rather than by guessing.

Free local embeddings with all MiniLM L6 v2

We use sentence-transformers with all-MiniLM-L6-v2: a small model that runs quickly on a normal CPU and outputs 384 dimensional vectors. It downloads from Hugging Face once, then works offline. No API key, no usage limits, and your notes stay on your laptop during indexing.

The key flag is normalize_embeddings=True, which scales every vector to length 1. You will see why that matters in a moment.

Always embed documents and questions with the same model. Vectors from different models are not comparable.

A numpy vector store and top k retrieval

A vector database, at its core, answers one question: which stored vectors point in the most similar direction to this query? The usual measure is cosine similarity:

cosine(a, b) = (a · b) / (|a| × |b|)

It is 1 for identical direction, around 0 for unrelated text. Because every vector already has length 1, the denominator is 1 and cosine similarity becomes a dot product. So search scores every chunk with one matrix multiplication, self.vectors @ q, and np.argsort(-scores)[:k] picks the k best. That is top k retrieval. Too small a k misses the right chunk; too large floods the prompt with noise. For notes, 3 to 5 is a good start.

Test retrieval before touching the LLM

If retrieval is wrong, no prompt can save the answer, so test it alone first:

from sentence_transformers import SentenceTransformer
from rag import EMBED_MODEL, VectorStore, chunk_pages

pages = [
    {"page": 1, "text": "Third normal form (3NF) removes transitive dependencies. " * 5},
    {"page": 2, "text": "The four Coffman conditions for deadlock are mutual exclusion, "
                       "hold and wait, no preemption and circular wait. " * 5},
    {"page": 3, "text": "UDP is connectionless and gives no delivery guarantee, unlike TCP. " * 5},
]
store = VectorStore(SentenceTransformer(EMBED_MODEL))
store.add(chunk_pages(pages, source="notes.pdf", chunk_size=400, overlap=80))
print(store.vectors.shape)  # (number_of_chunks, 384)

for q in ["What causes deadlock?", "is udp reliable", "Who won the IPL?"]:
    print(q, [(h["page"], round(h["score"], 2)) for h in store.search(q, k=2)])

Each exam question should retrieve its correct page first. Notice the off topic cricket question still returns something, just with a lower score. That drives a design decision in the app.

When to move to Chroma or FAISS

A numpy matrix is ideal for learning and for a few thousand chunks, which covers a semester of notes. Switch when you need:

• Persistence: chromadb saves the index to disk, so you do not re embed on restart, and stores metadata like page numbers.
• Scale: FAISS provides approximate nearest neighbour indexes for very large collections.
• Filters: queries such as "search only the OS notes".

Because VectorStore exposes only add and search, you can swap Chroma in later without touching the app.

A grounded prompt that cites sources and says "I don't know"

Retrieval finds the right chunks; the prompt decides whether the model uses them. Here is prompt.py:

SYSTEM_PROMPT = """You are StudyBuddy, a study assistant for a college student.
Answer ONLY using the numbered context passages provided.
Cite the passages you used like [1] or [2] after the sentence they support.
If the context does not contain the answer, say exactly:
"I don't know based on your notes." and suggest what to look up.
Do not use outside knowledge. Keep answers clear and exam friendly."""


def build_context(hits: list[dict]) -> str:
    blocks = []
    for n, h in enumerate(hits, start=1):
        blocks.append(f"[{n}] (source: {h['source']}, page {h['page']})\n{h['text']}")
    return "\n\n".join(blocks)


def build_messages(question: str, hits: list[dict]) -> list[dict]:
    context = build_context(hits)
    user = f"Context:\n{context}\n\nQuestion: {question}"
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user},
    ]

• Numbered passages with pages let the model cite [1], [2].
• "Answer ONLY using the context" stops the model blending general knowledge with your notes.
• An exact fallback sentence makes refusals consistent and easy to test.
• Rules in the system message, context in the user message makes it harder for odd text inside a PDF to override your instructions.

Grounding reduces hallucination but does not guarantee it. That is why the app also guards in code and shows the sources so the student can verify.

Adding it to the Streamlit app

Set LLM_BASE_URL to your provider's OpenAI compatible base URL (or your local Ollama server) and MODEL to a current model from their docs. Here is the new app.py:

import os

import streamlit as st
from dotenv import load_dotenv
from openai import OpenAI
from sentence_transformers import SentenceTransformer

from prompt import build_messages
from rag import EMBED_MODEL, VectorStore, chunk_pages, load_pdf

load_dotenv()
MODEL = os.environ.get("MODEL", "your-model-name")  # pick a current model from your provider's docs
TOP_K = 4
MIN_SCORE = 0.25  # below this, treat retrieval as "nothing relevant"

st.set_page_config(page_title="StudyBuddy")
st.title("StudyBuddy: chat with your notes")


@st.cache_resource
def get_client():
    return OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])


@st.cache_resource
def get_embedder():
    return SentenceTransformer(EMBED_MODEL)


@st.cache_resource(show_spinner="Reading and indexing your PDF...")
def build_store(file_bytes: bytes, file_name: str) -> VectorStore:
    import io
    pages = load_pdf(io.BytesIO(file_bytes))
    records = chunk_pages(pages, source=file_name, chunk_size=800, overlap=150)
    store = VectorStore(get_embedder())
    if records:
        store.add(records)
    return store


uploaded = st.sidebar.file_uploader("Upload your notes (PDF)", type="pdf")

if uploaded is None:
    st.info("Upload a PDF of your notes in the sidebar to begin.")
    st.stop()

store = build_store(uploaded.getvalue(), uploaded.name)
if not store.records:
    st.error("No text found. Is this a scanned PDF?")
    st.stop()
st.sidebar.success(f"Indexed {len(store.records)} chunks from {uploaded.name}")

if "messages" not in st.session_state:
    st.session_state.messages = []

for m in st.session_state.messages:
    with st.chat_message(m["role"]):
        st.markdown(m["content"])

question = st.chat_input("Ask something about your notes")
if question:
    st.session_state.messages.append({"role": "user", "content": question})
    with st.chat_message("user"):
        st.markdown(question)

    hits = [h for h in store.search(question, k=TOP_K) if h["score"] >= MIN_SCORE]
    with st.chat_message("assistant"):
        if not hits:
            answer = "I don't know based on your notes."
            st.markdown(answer)
        else:
            stream = get_client().chat.completions.create(
                model=MODEL, messages=build_messages(question, hits), stream=True
            )
            answer = st.write_stream(
                chunk.choices[0].delta.content or ""
                for chunk in stream
                if chunk.choices
            )
        if hits:
            with st.expander("Sources"):
                for n, h in enumerate(hits, start=1):
                    st.markdown(f"**[{n}]** page {h['page']}, score {h['score']:.2f}")
                    st.caption(h["text"][:300])
    st.session_state.messages.append({"role": "assistant", "content": answer})

What changed since Part 2

st.file_uploader accepts a PDF; getvalue() gives bytes that we wrap in io.BytesIO for pypdf.

st.cache_resource is the performance fix. Streamlit reruns the whole script on every interaction. Without caching, every message would reload the model and re embed the PDF. With it, the embedder and client load once, and build_store runs once per PDF because its arguments form the cache key: a new file triggers a fresh index, the same file is reused instantly. Use st.cache_resource for models, clients and indexes, and st.cache_data for plain data.

A minimum score guard. If no chunk scores above MIN_SCORE, the app replies "I don't know" without calling the LLM, saving a request and removing any chance of improvising. Treat 0.25 as a starting point; tune it by reading real scores in the Sources panel.

A Sources panel shows each retrieved chunk, its page and score. It is your best debugging tool.

Gotcha: follow ups like "explain the second one" are retrieved on their own, without earlier context. A common fix is one extra LLM call that rewrites the follow up into a standalone question before retrieval.

Run streamlit run app.py, then ask one question you know is in the PDF and one you know is not. Both behaviours matter.

Evaluating retrieval quality

Do not stop at "it seems to work". The simplest useful metric is hit rate at k: for questions where you know the answer's page, how often does that page appear in the top k results? Save as eval_retrieval.py:

from sentence_transformers import SentenceTransformer
from rag import EMBED_MODEL, VectorStore, chunk_pages, load_pdf

# (question, page that holds the answer), written by hand from your notes
TEST_SET = [
    ("Which conditions cause a deadlock?", 2),
    ("What does second normal form remove?", 1),
    ("Is UDP reliable?", 3),
]

store = VectorStore(SentenceTransformer(EMBED_MODEL))
store.add(chunk_pages(load_pdf("notes.pdf"), source="notes.pdf"))

k = 3
hits = 0
for question, page in TEST_SET:
    found = [h["page"] for h in store.search(question, k=k)]
    hits += page in found
    print("PASS" if page in found else "FAIL", question, found)
print(f"hit rate @{k}: {hits / len(TEST_SET):.0%}")

Write ten to twenty real questions, phrased casually and with different words from the notes. Then change one thing at a time (chunk_size, overlap, k, the embedding model) and keep the best setting. A crude word counting embedder we tried while testing failed precisely the question worded differently from the notes; a semantic model closes that gap, and this script makes such failures visible.

Then check generation by hand: is every claim supported by the cited chunk, do the [n] markers point to the right passages, and does an off topic question produce the fallback sentence?

Debug in order: retrieval first, prompt second, model last. Most "the AI is wrong" bugs in RAG are really "the right chunk was never retrieved" bugs.

For a structured path through Python and applied AI, see our courses.

Recap and next part

StudyBuddy now knows your notes. You learned why RAG usually beats fine tuning for facts, built every stage yourself (load, chunk, embed, store, retrieve, augment, generate), used overlap to protect ideas that cross boundaries, turned cosine similarity into one matrix product, wrote a prompt with citations and an honest "I don't know", cached the heavy work with st.cache_resource, and measured retrieval instead of guessing.

On your resume, name those engineering choices: chunk overlap, normalised embeddings, a score threshold and a retrieval evaluation. Run the bullet points through our AI resume review, and browse the jobs board to see where these skills appear.

In Part 4: Tools, function calling and agents, StudyBuddy learns to do things: call Python functions to generate quizzes, set revision reminders and decide for itself when to search your notes. Part 5: Deploy, secure and showcase then puts it online for recruiters to try.