Prompt Design for RAG Systems: What Goes in the Prompt vs What You Retrieve

RAG apps fail in a very specific, very frustrating way: you ship a "good" prompt, retrieval looks "fine," and the model still answers like it didn't read the docs you gave it.

The fix usually isn't "add more instructions." It's deciding what belongs in the prompt (stable behavior and decision rules) versus what belongs in retrieval (volatile knowledge and evidence). When we blur that boundary, we get bloated system prompts, noisy context windows, and answers that drift because the model is drowning in tokens.

Two research threads make this trade-off hard to ignore. First, prompt templates measurably change RAG quality and latency-sometimes by a lot-especially on smaller models [1]. Second, long contexts aren't "free": performance degrades when the model has to sift through too much irrelevant or poorly ordered material (the classic "lost in the middle" dynamic) [2]. A third, newer line of work adds a twist: if you let the model participate in retrieval decisions (agentic RAG), you can keep context smaller while improving accuracy-because the model learns to ask for exactly what it needs [3].

So here's the rule of thumb I use.

Your prompt is your operating system. Your retrieved context is your RAM.

---

The clean split: instructions are stable, knowledge is volatile

In a RAG system, the LLM receives one combined input: instructions + question + retrieved text. But conceptually, those components do different jobs.

The prompt should hold stable "how to behave" logic. Think policies, format, refusal rules, and the method for turning evidence into an answer.

The retrieved context should hold volatile "what is true right now" knowledge. Think product docs, policies, specs, tickets, emails-anything that changes or is too large to bake into the prompt.

This isn't just aesthetic. Pachtrachai et al. describe moving domain knowledge out of the system prompt and into the RAG store because long prompts dilute relevant instructions and can degrade utilization of the information you actually care about [2]. They explicitly call out the lost-in-the-middle effect and prompt length as practical reasons to keep the prompt lean while using retrieval for grounding.

My take: once you accept this split, prompt design gets easier. You stop arguing with yourself about whether to "just add that policy paragraph to the system prompt." Don't. Store it. Retrieve it when needed.

---

What goes in the prompt (and why)

Behavioral contract (always-on)

This is your non-negotiable behavior: role, tone, safety posture, and what "good" looks like. In RAG terms, it's the part you want cached, reused, and kept consistent across queries.

If you're building a support bot, this is where you define escalation behavior ("If the context doesn't contain the answer, say you don't know and ask a clarifying question"). If you're building an internal analyst, it's where you define standards ("No speculation; cite which source supports each claim").

Pachtrachai et al. show a progression from monolithic prompts to modular, governed prompts that stay internally consistent and avoid redundancy-specifically to improve reliability and portability across domains [2]. That's the direction I'd push any production RAG app: keep the prompt small enough that contradictions are obvious.

Evidence-use rules (the "grounding protocol")

This is the most underrated part. You want explicit rules for how the model should treat retrieved text, because retrieved text is messy: partial, duplicated, sometimes wrong, often conflicting.

A-RAG's paper makes this explicit at the system-prompt level: the agent should "ground your response in the retrieved documents," "cite the specific chunks," and "avoid speculation beyond what the documents support" [3]. Even if you're not building a tool-using agent, this is the right mental model: evidence is a constraint, not a suggestion.

The reasoning shape (but not the reasoning content)

The Mohammadi et al. study is basically a giant warning sign: prompts that force more elaborate reasoning structures can improve accuracy in multi-hop RAG, but often at big latency cost (8-10× in their setup) [1]. They also show something interesting: for a more capable model, a high-level "expert synthesis" instruction can outperform micromanaged step-by-step prompts on both accuracy and efficiency [1].

So I keep the prompt's "reasoning shape" lightweight. I'll ask for decomposition and synthesis, but I won't demand a verbose chain unless I'm in an offline workflow or using a smaller model that needs the scaffolding.

Output schema and verification hooks

If you need JSON, put it in the prompt. If you need citations, put it in the prompt. If you need the model to label uncertainty, put it in the prompt.

These are "interface contracts." They shouldn't live in retrieval because they're not knowledge-they're a protocol.

---

What you retrieve (and why)

Domain facts and policies

Anything that might change, differ by tenant, or be too long belongs in retrieval. That includes: pricing rules, refund windows, internal playbooks, API docs, and compliance text.

The RAG pipeline's job is to pull just enough of this for the answer. Not all of it.

Task-specific exemplars (sometimes)

I like retrieving examples when they are genuinely task-specific and brittle, like "here's the exact YAML config our platform expects" or "here's a known-good migration snippet." These are better as retrieved artifacts than prompt-embedded few-shots, because you can update them without touching your prompt, and you can route them per query.

"Just-in-time" definitions

For multi-hop questions, you often need one bridging fact (entity alias, acronym expansion, version mapping). That's retrieval territory. It's also where agentic RAG shines: the model can search at different granularities (keyword, semantic, chunk read) and stop when it has enough evidence [3].

---

The failure mode you're trying to prevent: context saturation

When people ask "prompt vs retrieval," they're often secretly asking: "Why did the model ignore the right sentence even though it was in context?"

A common culprit is too much context or poorly structured context. The more you stuff, the more you invite attention diffusion and positional weirdness. That's why the transcript-to-agent framework explicitly avoids embedding large domain knowledge in the system prompt, because excessive length dilutes utility [2].

And it's why agentic RAG is so appealing: it trades "stuff everything once" for "retrieve iteratively until sufficient" [3]. You're letting the model control context quantity and sequence, which is often what humans do naturally when researching.

---

Practical examples: a prompt template that respects the boundary

Here's a prompt skeleton I've used in production-style RAG. Notice how it contains behavior and protocol-but no domain facts.

System:
You are a reliable assistant for {company}. Your job is to answer using ONLY the provided sources.
If the sources do not contain the answer, say "I don't know based on the provided sources" and ask 1-2 clarifying questions.

Rules:
- Treat SOURCES as the only ground truth.
- If sources conflict, explain the conflict and ask what to follow (or prefer the newest policy if timestamps exist).
- Do not invent details, URLs, numbers, or API fields not present in sources.
- Write a short answer first, then details.
- Add citations like [S1], [S2] after the sentence they support.

User:
Question: {user_question}

SOURCES:
[S1] {chunk_1}
[S2] {chunk_2}
[S3] {chunk_3}

Now compare that to the temptation: embedding your whole policy manual in the prompt. Pachtrachai et al. basically show why that's a scalability dead-end: you get long prompts, weaker utilization, and more brittleness across domains [2].

If you want to go one step further toward agentic RAG, you can keep the same behavioral contract but expose tools (keyword search, semantic search, chunk read) and let the model pull more evidence when it's missing [3]. That's the "retrieve becomes part of prompting" twist that's starting to matter in 2026.

For a real-world "debugging mindset" example, the community post about a "semantic firewall" argues that many "prompt failures" are actually retrieval/context failures and suggests gating the model from answering when retrieved context is misaligned [4]. I don't treat that as research, but the instinct is right: separate "prompt quality" from "context quality," or you'll fix the wrong thing.

---

Closing thought

If you remember one thing, make it this: your prompt is not where you store knowledge. It's where you define how knowledge is used.

Keep the prompt stable, explicit, and short. Make retrieval do the heavy lifting. Then invest your effort where it pays: better chunking, better selection, better ordering, and (increasingly) letting the model retrieve iteratively when the question is genuinely multi-hop.

---

References

Documentation & Research

1. Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach - arXiv cs.CL
   https://arxiv.org/abs/2602.13890

2. From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants - arXiv cs.CL
   https://arxiv.org/abs/2602.15859

3. A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces - arXiv cs.CL
   https://arxiv.org/abs/2602.03442

4. Evolutionary Context Search for Automated Skill Acquisition - arXiv cs.LG
   https://arxiv.org/abs/2602.16113

Community Examples

5. A semantic firewall for RAG: 16 problems, 3 metrics, MIT open source - r/PromptEngineering
   https://www.reddit.com/r/PromptEngineering/comments/1r9z0c8/a_semantic_firewall_for_rag_16_problems_3_metrics/