RAG Systems Decoded: Chunking, Reranking, and Context Windows

Why GEO Practitioners Must Understand RAG

RAG is not an abstract computer science concept. It is the literal pipeline that decides whether your content gets cited by AI search engines. Every time a user asks ChatGPT, Perplexity, or Google AI Mode a question that triggers web search, a RAG pipeline executes. Your content either survives each stage of that pipeline or gets filtered out. Understanding where and why content gets dropped is the technical foundation of GEO.

According to a survey by Databricks, 87% of enterprise AI applications now use some form of RAG architecture (Databricks, 2025). This is not limited to consumer-facing AI search; enterprise AI tools, customer service bots, and internal knowledge systems all use RAG, making RAG optimization relevant across every AI touchpoint.

The Five Stages of a RAG Pipeline

Stage 1: Query Processing

The user's query is first processed and often expanded. For the query "best CRM for small business," the system might generate sub-queries like "CRM features comparison," "small business CRM pricing," and "CRM user reviews 2025." This query expansion step determines the breadth of documents retrieved. Research from Microsoft shows that query expansion increases retrieval recall by 23-40% depending on query complexity (Microsoft Research, 2025).

Stage 2: Document Retrieval

The expanded queries are run against the search index (Bing, Google, or proprietary). This stage uses a combination of sparse retrieval (keyword matching via BM25) and dense retrieval (semantic similarity via vector embeddings). According to research published at SIGIR 2025, hybrid sparse-dense retrieval outperforms either method alone by 18% on passage retrieval benchmarks (SIGIR, 2025).

The retrieval stage typically returns 20-100 candidate documents. Your content must be indexed and must match the query both lexically (keywords) and semantically (meaning) to survive this stage.

Stage 3: Document Chunking

Retrieved documents are broken into smaller passages, typically called "chunks." This is where content structure directly impacts your citation probability. RAG systems use several chunking strategies:

• Fixed-size chunking: Documents split every 200-500 tokens regardless of content boundaries. This is the simplest but least effective method.
• Semantic chunking: Documents split at natural boundaries like headings, paragraph breaks, and topic shifts. Research by LlamaIndex shows semantic chunking improves retrieval relevance by 31% compared to fixed-size chunking (LlamaIndex, 2025).
• Heading-based chunking: Sections defined by H2/H3 headings become individual chunks. This is the most common approach in production RAG systems and the reason heading structure matters so much for GEO.

Stage 4: Reranking

After chunking, a reranking model scores each chunk for relevance to the original query. Reranking models (like Cohere Rerank, BGE Reranker, or proprietary models) apply cross-attention between the query and each chunk, producing a relevance score. The top 5-15 chunks proceed to the generation stage.

Reranking is where specificity wins. Chunks containing exact query terms, specific data points, and direct answers score higher than general overviews. According to Cohere's benchmarks, chunks with factual claims and named entities rerank 42% higher than opinion-based or generic content (Cohere, 2025).

Stage 5: Context Window and Generation

The top-ranked chunks are inserted into the LLM's context window along with the system prompt and user query. The context window has a hard limit: GPT-4o supports 128K tokens, Claude 3.5 supports 200K tokens, and Gemini 1.5 Pro supports 1M tokens (OpenAI, 2025; Anthropic, 2025; Google, 2025). However, practical context usage is much smaller because processing more tokens increases latency and cost.

In practice, AI search engines typically insert 3,000-8,000 tokens of retrieved content into the context window. This means only 5-15 chunks from the entire web make it to the generation stage. Your content competes for these limited slots.

Practical Implications for Content Structure

Every stage of the RAG pipeline has direct implications for how you structure content:

• Query processing stage: Target both primary keywords (sparse retrieval) and semantic intent (dense retrieval). Include synonym variations and related concepts naturally in your content.
• Retrieval stage: Ensure indexation in all relevant search indexes (Google, Bing, proprietary). Use clear title tags and meta descriptions that match query patterns.
• Chunking stage: Write self-contained sections under clear headings. Each H2/H3 section should answer a complete sub-question without requiring context from other sections.
• Reranking stage: Include specific facts, statistics with sources, named entities, and direct answer statements at the beginning of each section. These are the signals reranking models weight highest.
• Context window: Keep sections concise (150-300 words per section). Longer sections risk being truncated or losing relevance density when competing for limited context window space.

The "Lost in the Middle" Problem

Research from Stanford and UC Berkeley demonstrated that LLMs exhibit a "lost in the middle" effect: information placed at the beginning and end of the context window is cited more frequently than information in the middle (Liu et al., 2024). In a RAG context, this means the order in which chunks appear in the context window affects citation probability.

While you cannot control chunk ordering directly, you can optimize for reranking score (which determines ordering). Chunks with the highest relevance scores appear first in the context window and benefit from the primacy effect.