Retrieval augmented generation is easy to demo and hard to operate. A notebook can retrieve a document and produce a grounded answer. A production system must keep indexes fresh, evaluate retrieval quality, handle partial failures and explain where answers came from.
The architecture below is a practical baseline for teams building on Google Cloud.
The production shape
The core pipeline has two paths: ingestion and query serving.
| Path | Components | Responsibility |
|---|---|---|
| Ingestion | Cloud Storage, Cloud Run Jobs, Document AI, Vertex AI embeddings | Convert source documents into indexed chunks |
| Retrieval | Vertex AI Vector Search, metadata filters | Find candidate context for a user question |
| Generation | Gemini on Vertex AI | Produce an answer with citations |
| Evaluation | BigQuery, scheduled jobs | Track retrieval quality over time |
Chunking is a product decision
Fixed-size chunking is tempting because it is simple. It also breaks tables, procedures and sections that should stay together. Better chunking preserves document structure and metadata.
For technical documentation, a useful default is section-aware chunking with overlap:
def chunk_section(section, max_tokens=700, overlap=120):
sentences = split_into_sentences(section.text)
chunks = []
current = []
for sentence in sentences:
if token_count(current + [sentence]) > max_tokens:
chunks.append(build_chunk(current, section.heading))
current = current[-overlap:]
current.append(sentence)
if current:
chunks.append(build_chunk(current, section.heading))
return chunks
Embeddings need measurement
Embedding choice should not be a brand preference. Build a small evaluation set: 50 to 100 real user questions and the documents that should be retrieved for each one. Then track precision at K and recall at K.
| Metric | Question it answers |
|---|---|
| Precision@5 | Are the top five chunks mostly useful? |
| Recall@10 | Did we retrieve the required source at all? |
| Citation coverage | Can the final answer point to sources? |
| No-answer accuracy | Does the system refuse when context is missing? |
Retrieval quality fails silently
A generation model can sound confident even when retrieval is bad. That is why retrieval metrics must be monitored separately from answer quality. If retrieval fails, the generation step should not hide it.
Operational guardrails
Production RAG needs the same delivery discipline as any other service:
- retries and circuit breakers around external model calls
- cache strategy for repeated queries
- token usage logging per tenant or feature
- source freshness checks
- explicit refusal behavior when evidence is weak
Final check
A production RAG system is not "LLM plus documents." It is a search system, an evaluation system and a generation system operating together. If retrieval cannot be measured, the answer cannot be trusted.