1. The Foundation of Retrieval Quality
In Retrieval-Augmented Generation (RAG) systems, the accuracy of the final LLM response is fundamentally constrained by the quality of the retrieved document chunks. If chunk sizes are too large, irrelevant noise dilutes the context and exceeds model context windows; if chunks are too small, critical surrounding context is severed, causing the LLM to hallucinate or miss key insights.
2. Fixed-Size Overlapping Chunking: Fast but Naive
The standard baseline approach splits text into fixed character or token counts (e.g., 500 tokens) with a sliding overlap window (e.g., 50 tokens). While computationally fast and simple to implement, fixed chunking frequently slices sentences in half or separates table headers from their underlying data rows, destroying semantic coherence.
3. Semantic and Document-Structure Aware Splitting
Advanced RAG pipelines respect document structural boundaries: Markdown and HTML chunkers split strictly along headers (H1, H2, H3) and paragraph breaks, preserving logical topical units. Semantic chunkers monitor embedding distance between consecutive sentences; when the mathematical similarity between adjacent sentences drops sharply, the chunker places a boundary, ensuring each chunk covers a single distinct thought.
4. Parent-Child Hierarchical Indexing
The gold standard for complex technical documentation is the Parent-Child retrieval pattern: large "Parent" chunks (e.g., 1500 tokens) are subdivided into multiple small "Child" chunks (e.g., 200 tokens). Vector search indexes and matches against the granular child embeddings; however, when a match occurs, the system retrieves and passes the broader parent chunk to the LLM prompt, delivering pinpoint precision with complete contextual awareness.
