Scaling Retrieval-Augmented Generation for Enterprise Workloads

Why the old pull‑model feels brittle

For years many teams built question‑answering services by loading the entire knowledge base into a single large language model prompt. The approach works for a few hundred documents, but once the corpus grew beyond a few megabytes the latency spiked, memory errors appeared, and updates became a nightmare. The whole system would stall whenever a new policy document was added, and the team spent more time shuffling text than delivering value.

Switching to a retrieval‑augmented generation (RAG) pipeline promised to break that wall: keep the model small, let a fast vector store fetch the most relevant chunks, and let the model focus on synthesis. In theory the architecture scales linearly with the number of documents, but in practice the first production rollout hit a wall that wasn’t obvious from the design docs.

Background – why enterprises care now

Enterprises are stuffing their internal wikis, compliance manuals, and product specifications into AI assistants. The promise is a single endpoint that can answer any employee query with up‑to‑date facts. The stakes are high: a wrong answer can lead to compliance breaches or costly rework. That pressure forces teams to move from prototype to production quickly, often with limited ops bandwidth.

In my own work at AkkisTech we saw three patterns emerge: (1) the vector store was treated as a black box, (2) the document chunking strategy was static, and (3) the orchestration layer assumed the retrieval step would always finish under a second. When any of those assumptions broke, the whole request pipeline stalled.

Technical deep dive – from flaky prototype to stable service

The core of the architecture consists of four parts:

• Ingestion pipeline that splits raw PDFs, HTML, and Markdown into semantic chunks.

• Embedding service that turns each chunk into a dense vector using a transformer encoder (e.g., sentence‑transformers 2.2.2).

• Vector database (we use Milvus 2.3.0) that stores vectors with metadata and provides k‑nearest‑neighbor search.

• Generation service that receives the top‑k chunks, builds a prompt, and calls the LLM (GPT‑NeoX 20B on a GPU‑accelerated inference server).

The problem appeared after the ingestion pipeline was updated to include new policy documents. Latency jumped from ~800 ms to >4 seconds, and occasional time‑outs started surfacing. The first clue was the Milvus logs: they reported “search timeout” and “index rebuild in progress”.

Investigation revealed two intertwined issues:

1. Chunk size was set to a fixed 500 tokens. New documents contained long tables and code blocks, which produced many tiny chunks. The vector count exploded, and Milvus had to scan a much larger index for each query.

2. The embedding service was running in a single‑threaded FastAPI process. When the ingestion job flooded the queue with new vectors, the service became a bottleneck, delaying the background index refresh.

Fixing the first issue required a dynamic chunker. The new logic uses a sliding window with a maximum of 300 tokens but also respects semantic breakpoints (headings, list items). Here’s a trimmed version of the implementation:

def chunk_document(text: str, max_tokens: int = 300) -> List[str]:
    """Split *text* into chunks that are  max_tokens:
            # Very long sentence – split it aggressively
            parts = [s[i:i+max_tokens] for i in range(0, len(s), max_tokens)]
            for p in parts:
                chunks.append(p)
            continue
        if current_len + token_len <= max_tokens:
            current.append(s)
            current_len += token_len
        else:
            chunks.append(" ".join(current))
            current = [s]
            current_len = token_len
    if current:
        chunks.append(" ".join(current))
    return chunks

After redeploying the chunker, the vector count dropped by 40 % and Milvus reported stable search times.

The second issue was solved by moving the embedding service into a Celery worker pool with three concurrent processes, each bound to its own GPU slice. The updated Docker‑Compose snippet looks like this:

services:
  embed-worker:
    image: myorg/embedding-service:2.0
    deploy:
      replicas: 3
    environment:
      - CUDA_VISIBLE_DEVICES=0,1,2
    command: celery -A embed_worker worker --concurrency=1

With parallel workers, the ingestion backlog cleared within minutes, and Milvus could finish its index rebuild without hitting the timeout threshold.

Common pitfalls – what tends to go wrong

Even after the fixes above, teams still stumble over a few recurring mistakes:

• Ignoring vector drift. When document revisions are frequent, old vectors linger in the database, polluting the nearest‑neighbor results. A periodic re‑embedding job that purges stale IDs is essential.

• Hard‑coding k‑value. Picking a static top‑k (e.g., 5) works for small corpora but under‑represents larger knowledge bases. A dynamic approach that adjusts k based on similarity scores yields more reliable answers.

• Skipping prompt sanitisation. User queries may contain special characters that break the LLM’s JSON parser. Simple escaping or a whitelist of allowed characters prevents hidden crashes.

• Under‑estimating latency variance. Network hops between the API gateway, vector store, and inference server add jitter. Monitoring the 95th percentile latency and setting appropriate timeout margins helps keep the SLA stable.

Practical implementation guide – step by step

Below is a checklist that can turn a shaky prototype into a production‑ready RAG service:

1. Define a chunking policy. Start with a max token limit of 300, enable semantic break detection, and write unit tests that verify edge cases (long tables, code blocks).

2. Choose an embedding model with known performance. Sentence‑transformers all‑mini‑lm‑l6‑v2 (2.2.2) runs comfortably on a single RTX 3090 and yields ~0.12 seconds per chunk.

3. Provision a vector database that supports incremental indexing. Milvus 2.3.0 with IVF_PQ index provides sub‑millisecond search for up to 10 million vectors.

4. Wrap the embedding step in an async job queue. Celery + Redis works well; configure three workers per GPU and set a retry policy for transient failures.

5. Implement a re‑embedding schedule. Use a cron job that scans for documents modified in the last 24 hours, deletes old vectors, and inserts fresh ones.

6. Build a thin orchestration layer. A FastAPI endpoint that:

• receives the user query,

• calls the vector store with a dynamic k based on similarity threshold,

• assembles the prompt,

• invokes the LLM, and

• returns the answer with source citations.

7. Set observability hooks. Export Prometheus metrics for ingestion lag, search latency, and LLM response time. Alert on 95th percentile spikes.

8. Run a load test. Simulate 200 RPS with mixed query lengths; watch for queue back‑pressure and adjust worker count accordingly.

Following this roadmap usually eliminates the most common stability issues and gives you a clear path to scale horizontally when request volume grows.

Closing thoughts – keep the system simple and observable

RAG feels like a magical shortcut, but the underlying pieces still need careful engineering. The biggest lesson from the postmortem is that every component – chunker, embedder, vector store, and generator – should expose metrics that tell you when it’s straining. When you see a sudden jump in search latency, the first places to look are the index size and the ingestion backlog.

If you keep the chunking logic adaptable, run embeddings in parallel, and treat the vector store as a mutable cache rather than a write‑once store, the architecture will stay responsive even as the document corpus balloons to millions of pages.

Next time you add a new compliance handbook, remember to trigger a re‑embedding job and watch the index health dashboard. The extra step saves you from a cascade of time‑outs later on.

Ready to try it out?

Grab the starter repo at [Feel free to reach out at kerimakkis.com if you want to discuss this further.

If you found this useful, check out my other articles and projects at kerimakkis.com. I write about full-stack development, AI integration, and the tools I actually use in production.

Share on LinkedIn