Semantic Chunking: How It Works in RAG and AI Search

Semantic chunking is a way of splitting text for retrieval-augmented generation (RAG) by meaning instead of by length. It breaks a document into sentences, turns each one into an embedding, and starts a new chunk wherever the similarity between neighboring sentences drops sharply. The result is a set of chunks that each hold one topic, which makes them easier to retrieve and quote accurately.

This guide explains how semantic chunking works, how it compares with fixed-size, recursive, sentence and agentic chunking, and what the research says about when it is worth the extra cost. It includes a short Python example, then covers what chunking means for writers: how to format paragraphs and headings so AI search systems can lift your passages cleanly.

What is semantic chunking?

Semantic chunking is a document-splitting strategy that places chunk boundaries where the topic changes, detected by comparing the embeddings of consecutive sentences. Fixed-size methods cut every N tokens. Semantic chunking cuts where meaning shifts.

In a RAG pipeline, documents are split into chunks, each chunk is embedded as a vector, and the chunks most similar to a user's question are retrieved and passed to a language model. The chunk is the unit that gets found, ranked and quoted. If a chunk mixes two topics, its embedding is a blur of both, and it matches neither question well.

The technique was popularized by Greg Kamradt's "5 Levels of Text Splitting" tutorial. Both major RAG frameworks credit him: the LangChain SemanticChunker source code says the method was "taken from Greg Kamradt's wonderful notebook," and the LlamaIndex semantic chunker documentation describes its module as an adaptation of the same idea.

How does semantic chunking work?

Semantic chunking works in four steps: split into sentences, embed each sentence with its neighbors, measure the distance between consecutive embeddings, and break wherever that distance is unusually large.

1. Split into sentences. The text is broken into sentences with a regex or a tokenizer such as NLTK or spaCy. LangChain's default regex splits after a period, question mark or exclamation point followed by whitespace.
2. Embed with a buffer. Each sentence is combined with the sentences around it (a "buffer," default 1 on each side in LangChain) and embedded. The buffer smooths out noise from very short sentences.
3. Measure cosine distance. The system calculates the cosine distance between each sentence group and the next one. A small distance means the text is still on the same topic. A spike means the topic has changed.
4. Pick breakpoints. Any distance above a threshold becomes a chunk boundary. Sentences between two boundaries are joined into one chunk.

The threshold is where implementations differ. LangChain's SemanticChunker supports four breakpoint_threshold_type options, each with a default you can override.

LlamaIndex's SemanticSplitterNodeParser exposes the same idea through buffer_size and breakpoint_percentile_threshold, set to 1 and 95 in its documented example. Its docs note the sentence-splitting regex works mainly for English.

Semantic chunking example in Python

The clearest way to understand semantic chunking is to build a minimal version. The function below follows the same logic as LangChain's implementation using an open-source embedding model.

1# pip install sentence-transformers numpy2import re3import numpy as np4from sentence_transformers import SentenceTransformer5 6def semantic_chunks(text, percentile=95, buffer=1, model_name="all-MiniLM-L6-v2"):7    # 1. Split into sentences8    sentences = [s for s in re.split(r"(?<=[.?!])\s+", text.strip()) if s]9    if len(sentences) < 3:10        return [" ".join(sentences)]11 12    # 2. Embed each sentence with its neighbors to smooth out noise13    windows = [14        " ".join(sentences[max(0, i - buffer): i + buffer + 1])15        for i in range(len(sentences))16    ]17    model = SentenceTransformer(model_name)18    emb = model.encode(windows, normalize_embeddings=True)19 20    # 3. Cosine distance between each window and the next one21    distances = 1 - np.sum(emb[:-1] * emb[1:], axis=1)22 23    # 4. Split wherever the distance is above the chosen percentile24    threshold = np.percentile(distances, percentile)25    chunks, start = [], 026    for i, distance in enumerate(distances):27        if distance > threshold:28            chunks.append(" ".join(sentences[start: i + 1]))29            start = i + 130    chunks.append(" ".join(sentences[start:]))31    return chunks32 33text = (34    "Coffee beans are roasted. A light roast keeps acidity. "35    "Dark roast coffee tastes bitter. Income tax is due in April. "36    "You file income tax online. Late tax filings cost money."37)38for chunk in semantic_chunks(text, percentile=80):39    print("-", chunk)

On a short sample like this, lower the percentile so the function has room to split. The two chunks it returns separate the coffee sentences from the tax sentences, which is exactly what a fixed 100-character splitter would fail to do.

In production, most teams use the framework version instead of rolling their own:

1# pip install langchain-experimental langchain-openai2from langchain_experimental.text_splitter import SemanticChunker3from langchain_openai import OpenAIEmbeddings4 5splitter = SemanticChunker(6    OpenAIEmbeddings(),7    breakpoint_threshold_type="percentile",8    breakpoint_threshold_amount=95,9)10docs = splitter.create_documents([long_text])

The same class also accepts number_of_chunks if you want a target count, and min_chunk_size to stop it from producing tiny fragments.

Semantic chunking vs fixed-size, recursive, sentence and agentic chunking

Semantic chunking is one of five common approaches, and it is not always the best one. Fixed-size and recursive splitters are cheaper, structure-aware splitters exploit headings, and agentic (LLM-driven) chunking can find better boundaries at a much higher cost.

Recursive chunking vs semantic chunking

Recursive chunking splits on formatting (paragraph breaks, then line breaks, then sentences), while semantic chunking splits on meaning. If your documents already have clean paragraphs and headings, recursive splitting gets most of the benefit for a fraction of the compute.

Semantic chunking earns its cost on text with weak structure: transcripts, emails, scraped web pages, long unbroken paragraphs. A common pattern is to split by headings first and apply semantic splitting only to sections that are still too long.

What is agentic chunking?

Agentic chunking hands the boundary decision to a language model. LumberChunker (Duarte et al., 2024) repeatedly prompts an LLM to find the point in a group of passages where the content starts to shift, and reports a 7.37% improvement in retrieval (DCG@20) over the strongest baseline on long narrative documents.

Is semantic chunking worth it? What the research says

Semantic chunking is not a guaranteed upgrade. Independent evaluations show it can help on some corpora and lose to a well-tuned recursive splitter on others.

• Mixed results on benchmarks. In Is Semantic Chunking Worth the Computational Cost?, Qu, Tu and Bao (2024) tested document retrieval, evidence retrieval and answer generation and concluded that the extra compute was "not justified by consistent performance gains" over fixed-size chunking.
• Defaults matter more than labels. Chroma's Evaluating Chunking Strategies for Retrieval report (July 2024) found recall varied by up to 9 points across strategies. A recursive splitter at 200 tokens with no overlap performed consistently well, while the Kamradt-style semantic chunker at default settings scored 83.6% recall, below most alternatives. Chroma's own cluster-based semantic chunker and an LLM-based chunker posted the best precision and recall respectively.
• Context can beat boundaries. Anthropic's contextual retrieval method prepends a short, document-aware summary to each chunk before embedding. It cut failed top-20 retrievals by 35% on its own and by 67% when combined with BM25 and reranking.

The practical takeaway: chunk boundaries matter, but so do chunk size, overlap, the embedding model and whether each chunk carries enough context to stand alone. Evaluate on your own queries before committing to any method.

Best practices for semantic chunking in a RAG pipeline

The strongest pipelines combine structure, meaning and context rather than relying on one splitter.

1. Split on structure first. Use headings, sections and HTML or Markdown elements as hard boundaries, then apply semantic splitting inside long sections.
2. Cap chunk size. Semantic chunkers can produce very long chunks when a section never changes topic. Set a maximum token limit and a min_chunk_size to avoid fragments.
3. Tune the threshold on real queries. Start at the 95th percentile, then adjust while measuring recall and precision on a labeled set of questions from your own domain.
4. Add context to each chunk. Prepend the document title and section heading, or a generated summary, so a chunk that says "it" or "the company" still makes sense alone.
5. Keep metadata. Store the source URL, heading path and position with each chunk so answers can cite precisely.
6. Re-evaluate when you change embedding models. Distance distributions shift between models, so a threshold tuned for one may over-split or under-split with another.

What semantic chunking means for content and AI search

For Google Search specifically, you do not need to chunk your content: Google's guide to optimizing your website for generative AI features (last updated 10 July 2026) lists breaking content into small chunks among the things you can ignore, alongside llms.txt and special schema. Do not split pages into fragments for Google's sake.

Self-contained passages still help elsewhere. ChatGPT, Perplexity, enterprise RAG tools and other retrieval systems split pages with their own chunkers, and you cannot control how. Writing so that almost any reasonable splitter produces clean, standalone passages is good editing that also suits those systems.

Google has said for years that it ranks passages, not only pages. Its October 2020 passage ranking announcement said the change would affect about 7% of queries by helping surface a single relevant passage buried deep in a long page. Google also says its AI features may issue multiple related searches across subtopics, with no special markup or AI text files needed.

That fan-out behavior is one reason clear passages matter: a sub-question can be answered by one section of your page rather than the whole thing. Our guide to query fan-out in AI search explains that side of the process.

The chunk paragraph format: how to write self-contained passages

A chunk-friendly paragraph states one idea, names its subject explicitly, and makes sense if it is the only thing a reader sees. Three sentences or fewer is a good rule.

Hard to retrieve: "As mentioned above, this approach has a few drawbacks. It costs more, and it isn't always better. That's why many teams skip it."

Easy to retrieve: "Semantic chunking costs more than fixed-size chunking because every sentence must be embedded at ingestion time. Benchmarks show it does not always improve retrieval. Many teams start with recursive splitting and add semantic splitting only for unstructured text."

The second version repeats the subject, carries its own context and contains a claim that can be quoted. It is longer by a few words and far more useful to a retrieval system.

Usability research points the same way. Nielsen Norman Group's guide to chunking content recommends short paragraphs, clear visual hierarchy, contrasting headings and lists so readers can scan and remember information. What helps a skimming human also helps a splitter.

Heading structure that produces clean chunks

Headings are the boundaries most splitters respect first, so treat each H2 or H3 as the start of a standalone answer.

• Phrase headings as the question or topic a searcher would type ("How does semantic chunking work?" rather than "The mechanics").
• Answer the heading in the first one or two sentences under it.
• Keep one topic per section; if a section covers two ideas, give each its own H3.
• Avoid references that depend on earlier sections, such as "as we saw above" or "this method."
• Put key facts in HTML text, not only in images or scripts. Many AI crawlers do not render JavaScript, so content injected client-side may never be chunked at all.

Tables and lists also chunk well when they have a caption and clear column labels. Our guide to optimizing tables, PDFs and infographics for AI citation covers non-text formats in detail.

A checklist for chunk-ready content

Run this check on any page you want AI systems to cite:

1. Does every H2 and H3 describe one specific question or topic?
2. Does the first sentence under each heading answer it directly?
3. Can each paragraph be understood if it is read alone?
4. Does each paragraph name its subject instead of relying on "it" or "this"?
5. Are numbers and claims attributed to a named source in the same sentence?
6. Is the key content in server-rendered HTML that crawlers can read?
7. Do comparison and step content use tables or numbered lists?

If a page fails three or more of these, it usually needs a structural rewrite rather than a light edit. Our content refresh guide explains how to prioritize which pages to fix first, and the AI search readiness audit covers the technical checks around it.

Check access before structure: Clean chunks only help if crawlers can reach the page. Lantern's free AI Crawler Checker shows which AI crawlers your robots.txt allows and the rule behind each decision.

How to tell whether your passages are being retrieved

The only reliable test is to look at AI answers themselves: which of your pages get cited, for which questions, and which competitor passages appear instead.

Lantern's research shows why the page type matters as much as the prose. In Beyond the Homepage, the brand with the broadest owned footprint saw ChatGPT cite its editorial pages most often (38.1% of owned citations), while Google AI cited product pages most (53.6%). Focused pages that answer one question well gave AI systems better evidence than a homepage trying to cover everything.

Lantern tracks citations and mentions across ChatGPT, Google AI Overviews, Gemini, Perplexity and Claude, and shows which pages and sources each engine uses. The free AI Visibility Report is a quick way to see where you stand before restructuring content. For the metrics behind it, see what AI search visibility is and how it's measured.

If you write the way retrieval systems read, one idea per passage, a clear heading over each answer and enough context to stand alone, your content is easier to find for RAG pipelines and AI search engines alike. Semantic chunking is how machines find those boundaries. Good structure is how you make sure they find the right ones.

Frequently asked questions

What is semantic chunking in simple terms?

Semantic chunking means cutting a long document into pieces at the points where the topic changes, instead of every few hundred words. A computer finds those points by checking how similar each sentence is to the next. Each resulting chunk covers one idea, so it is easier to search and quote.

What is a semantic chunker?

A semantic chunker is the software component that performs semantic chunking. Common examples are SemanticChunker in LangChain's experimental package and SemanticSplitterNodeParser in LlamaIndex. Both take an embedding model and a threshold setting and return chunks split by meaning.

Is semantic chunking better than recursive chunking?

Not always. Recursive chunking is cheaper and performs well on documents with clean paragraphs and headings, and some benchmarks found it matched or beat semantic chunking. Semantic chunking tends to help most on unstructured text such as transcripts, emails or scraped pages.

What chunk size should I use for RAG?

There is no universal best size. Chroma's evaluation found a 200-token recursive split with no overlap performed consistently well, while large chunks with heavy overlap lowered precision. Test a few sizes against real questions from your domain and measure recall and precision.

Does Google use semantic chunking?

Google has not published how it splits pages for AI Overviews or AI Mode, and it says site owners do not need to chunk content for Google Search. It has confirmed that it ranks individual passages and that AI features can issue multiple related searches. Clear headings and self-contained passages are good writing for Google and help other retrieval systems.

What is the difference between content chunking and semantic chunking?

Content chunking is a writing and UX practice: breaking information into short, scannable units for readers. Semantic chunking is a machine process that splits text by meaning for retrieval. They reinforce each other, because well-chunked content gives a semantic chunker clean boundaries to find.