Building Robust Retrieval‑Augmented Generation Pipelines

Why RAG pipelines flop when you rush the first step

Most teams jump straight to prompting a language model with a raw document dump. The result is a flood of irrelevant answers, slow response times, and a sudden spike in cost. The root cause is almost always the same: data was never sliced into searchable pieces, the vector index was built without thinking about distance metrics, and the prompt never accounted for the retrieval step. The model ends up hallucinating because it cannot find the right context fast enough. This opening mistake sets the tone for the whole system – everything that follows inherits the noise.

Why the timing is right for a disciplined RAG approach

Enterprises are now willing to pay for domain‑specific reasoning. Large language models have become cheap enough to run on‑prem, yet the cost of unnecessary API calls still hurts the bottom line. At the same time, vector databases have matured: open‑source options like Chroma or Milvus sit comfortably beside managed services such as Pinecone or AWS OpenSearch. The sweet spot is a pipeline that knows when to stay local for latency‑sensitive queries and when to reach out to the cloud for heavy‑weight embeddings. Senior developers can extract that balance only by treating each stage as a separate decision point rather than a single monolith.

Lesson 1 – Chunking is not just splitting text

When you break a PDF into paragraphs you assume each paragraph is a self‑contained fact. In practice the semantic boundary often crosses sentence borders, especially in technical manuals. A robust chunker therefore combines three heuristics:

• Maximum token count (usually 300‑400 for models like Llama‑2‑13B)
• Overlap of 50‑100 tokens to preserve context across windows
• Semantic break points using a lightweight sentence‑embedding model (for example MiniLM‑v2) to avoid cutting mid‑concept

Below is a minimal Python snippet that demonstrates these ideas without pulling in a heavyweight library:

from typing import List

def chunk_text(text: str, max_tokens: int = 350, overlap: int = 75) -> List[str]:
    # naive tokeniser – split on whitespace
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        end = i + max_tokens
        chunk = ' '.join(words[i:end])
        chunks.append(chunk)
        i = end - overlap  # step back for overlap
    return chunks

# example usage
sample = """Large language models excel at reasoning when they have the right context. However, feeding them an entire knowledge base at once defeats the purpose. By chunking we keep each request small and focused."""
print(chunk_text(sample))

The trade‑off here is obvious: more overlap means larger index size and higher latency, but it also reduces the chance of cutting a key phrase. In production you would replace the naive split with a tokenizer that respects the model’s token encoding.

Lesson 2 – Picking the right vector store for your workload

Local versus cloud is a classic tug‑of‑war. A local store (e.g., Chroma running in a Docker container) gives you sub‑millisecond lookup for small collections, no network jitter, and full control over hardware. The downside is that you must provision RAM and SSD yourself, and scaling beyond a few million vectors becomes painful.

Managed services shine when you need to serve billions of vectors or you want automatic replication across zones. They usually charge per million vectors and per query, so cost can climb quickly if your query volume spikes. Latency on a well‑tuned managed index is still in the low‑double‑digit milliseconds, but you lose the ability to tweak the distance metric beyond what the service offers.

My rule of thumb: start local for proof‑of‑concepts, then migrate to cloud only after you have measured query patterns. If your average query touches under 10 k vectors, a local store will stay under a dollar a day on a modest VM. If you exceed that, evaluate a managed option and watch the cost dashboard closely.

Lesson 3 – Embedding models: local quantised vs cloud API

Running embeddings on‑prem gives you predictable latency and no per‑token cost. Quantised models such as GPT‑Neo‑X‑Quant can embed a 300‑token chunk in about 30 ms on a single GPU. The trade‑off is slightly lower quality – the cosine similarity scores are noisier, which can increase the chance of retrieving irrelevant chunks.

Cloud APIs (e.g., OpenAI embeddings) deliver state‑of‑the‑art quality with a single HTTP call. The price is usually $0.0001 per 1 k tokens, and latency can vary between 100‑300 ms depending on network conditions. For low‑traffic internal tools the cost is negligible, but for a public‑facing chatbot the per‑query price adds up fast.

When you blend both, you can use a cheap local model for the first‑pass filter (e.g., top‑10 candidates) and then re‑rank those with a cloud model. This hybrid approach caps cost while preserving relevance.

Lesson 4 – Prompt tuning to keep hallucinations in check

Even with perfect retrieval, a language model can invent facts if the prompt does not force grounding. A reliable pattern is the "retrieval‑augmented" template:

Answer the question using only the provided context. If the answer is not present, say "I don't know".

Context:
{retrieved_chunks}

Question: {user_question}
Answer:

This template does two things:

• It explicitly tells the model to stay within the supplied text.
• It gives a fallback phrase that you can detect programmatically.

Experiment with a few variations – adding "Cite the source after each sentence" or limiting the answer length. In my experience, a 2‑sentence limit reduces the chance of the model drifting into unrelated territory, especially when the retrieved chunks are short.

Common pitfalls that bite even seasoned engineers

1. Forgetting to refresh the vector index after a data update. The index becomes stale, leading to missed facts and higher hallucination rates. Schedule a re‑index job that runs nightly or trigger it on every commit if the knowledge base changes frequently.

2. Ignoring token limits in the final prompt. If you concatenate too many chunks, the request exceeds the model’s context window and the API truncates silently. A safe guard is to compute the total token count before sending and drop the lowest‑scoring chunks until you fit.

3. Relying on a single distance metric. Cosine works well for most embeddings, but for dense vectors with a lot of zeroes, inner product can surface better matches. Test both on a validation set before locking in.

4. Not monitoring hallucination signals. Simple heuristics like “answer contains words not found in any chunk” or “confidence score below a threshold” can trigger a fallback path. Without this, you’ll only notice problems after users complain.

Step‑by‑step guide to a production‑grade RAG pipeline

1. Gather source documents (markdown, PDFs, API responses). Store them in a version‑controlled bucket.
2. Run a preprocessing script that extracts raw text, normalises whitespace, and strips boilerplate.
3. Apply the chunker from Lesson 1, storing each chunk with a unique ID and a reference back to the source file.
4. Choose an embedding model. For a local start, install a quantised MiniLM‑v2 checkpoint. Generate embeddings for every chunk and persist them in a local Chroma collection.
5. If you need scale, export the embeddings to a CSV and import them into a managed service. Keep the same dimensionality to avoid re‑training.
6. Build a retrieval wrapper that takes a user query, embeds it with the same model, performs a nearest‑neighbour search (top k = 5 or 10), and returns the raw text of the chunks.
7. Compose the prompt using the template from Lesson 4, injecting the retrieved chunks and the user question.
8. Send the prompt to your LLM endpoint (local server or cloud API). Capture the raw answer and the “I don’t know” flag.
9. Post‑process: if the flag is set, log the query for future knowledge‑base enrichment. Otherwise, render the answer to the user, optionally attaching source citations.
10. Instrument metrics – latency per stage, cost per query, and hallucination rate (percentage of answers that contain out‑of‑context phrases). Set alerts if any metric crosses a threshold.

This framework lets you swap components (e.g., replace the embedding model) without rewriting the whole flow. Each step is a clear contract: input, output, and performance expectations.

Closing thoughts – keep the loop tight

The most reliable RAG pipelines are the ones that treat data, retrieval, and generation as a feedback loop. When you notice a spike in hallucination rate, dig into the retrieval scores first – a bad match is often the culprit. If retrieval looks solid, examine the prompt; a missing “cite your source” clause can let the model wander.

Remember that cost, latency, and relevance are three sides of the same triangle. Optimising one side will usually move the other two. The key is to measure continuously and to keep the architecture modular so you can pivot without a full rewrite.

Want to see this in action?

Head over to Feel free to reach out at kerimakkis.com if you want to discuss this further.

If you found this useful, check out my other articles and projects at kerimakkis.com. I write about full-stack development, AI integration, and the tools I actually use in production.

Share on LinkedIn