Design a RAG question-answering service

Question 1

Who is asking, and what comes back?

Accepted Answer

A question in, and an answer streamed back with the passages it was written from. Citations are not decoration here — they are the only way a reader can check an answer that is fluent whether or not it is true.

Question 2

What receives the question and decides who is allowed to ask it?

Accepted Answer

A service in front of everything else: it authenticates, resolves which documents this user may see, and streams the answer. Identity has to be established here because retrieval two hops later needs it as a filter.

Question 3

How does a 40-page document become something you can search?

Accepted Answer

It is split into chunks of a few hundred tokens on section boundaries, each chunk is embedded, and the vector is written to the index with the text beside it. This is an offline batch job, and it is where the quality of every answer is actually decided — a chunker that splits mid-argument cannot be rescued by anything downstream.

Question 4

Where do the embedded chunks live, and how big is that?

Accepted Answer

A vector index, and it is worth sizing out loud: 10M chunks at 1,536 float32 dimensions is 6 KB each, about 60 GB of vectors and half as much again for the graph. That is one large machine, not a cluster — and int8 quantisation cuts it fourfold for a point or two of recall.

Question 5

Fifty candidates come back from the index. Which three go in the prompt?

Accepted Answer

Retrieve wide and narrow late: a cheap ANN plus keyword search for candidates, then a cross-encoder reranker that scores the question against each one jointly. Skipping the rerank asks the cheapest component in the system to make its most consequential decision.

Question 6

What writes the answer, and what is in the prompt?

Accepted Answer

The model, given the question and the top few passages. It is the slowest and by far the most expensive box on the board — three thousand input tokens and three hundred out, two million times a day, is where the entire budget of this system goes.

Question 7

The same question gets asked all day. What stops you paying for it twice?

Accepted Answer

A semantic cache in front of retrieval and generation both, keyed on the question’s embedding rather than its exact text. It is the highest-leverage component here and the most dangerous: two questions can be neighbours in vector space and have different answers, so the threshold is tight and the key is scoped per tenant.

Question 8

Your client calls the model directly. What did the answer lose?

Accepted Answer

Everything that made it a RAG system: retrieval, the permission filter, the citations, the cost controls and the log of what the model was shown. A client that can reach the model can also spend your token budget on anything it likes.

Question 9

How many questions a second is 2M a day?

Accepted Answer

2,000,000 / 86,400 is about 23 a second — nothing, as a request rate. The number that matters is the one underneath it: roughly 3,300 tokens per question is about 76,000 tokens a second, and tokens are what you are billed in.

Question 10

This system can be badly wrong without a single error being logged. How would you know?

Accepted Answer

Because the output is a fluent paragraph whether or not it is right, nothing in latency or error rate catches a bad chunker, a stale index or an embedding model that was upgraded underneath you. The answer is an offline eval set run in CI — retrieval recall separately from answer quality, so a search bug and a generation bug are distinguishable — plus a log of the exact chunks every answer was built from.

Devscape — system design