RAG in Practice: Does Chunk Size Change the Answer?

Ask any AI assistant how to split documents for RAG and you will get a similar answer: chunks of 500 to 1,000 tokens, with 10% to 20% overlap.

It is a plausible answer. It shows up in hundreds of tutorials. And almost nobody shows whether it works.

So we tested it. We took a real law, wrote 32 questions with known answers and compared seven ways of cutting the text, in English and in Portuguese.

The standard recipe landed near the bottom. The best strategy found more answers and sent four times less text to the LLM.

The full code, with the data and the questions, is at salada-dados/blog. Everything runs for free, on your own machine, with no API key.

RAG in one sentence

RAG (Retrieval-Augmented Generation) finds the right passages in a document and only then asks the LLM to answer, using those passages.

In practice, it takes four steps:

1. Cut the document into pieces (chunks).
2. Turn each chunk into a vector (embedding) and store it in a database.
3. When a question arrives, turn it into a vector and search for the most similar chunks.
4. Send those chunks to the LLM, along with the question.

If step 3 brings back the wrong passage, the LLM will answer confidently, but from the wrong text. That is why this article measures only the search: without the right passage, no LLM can save the answer.

And the search depends heavily on step 1, which is usually decided by guesswork.

The document: the GDPR

We used the GDPR, the European data protection regulation that became a model for laws like Brazil's LGPD. It makes a good test for three reasons:

• It is the size of a real document: 99 articles, 389 numbered paragraphs, 40 to 45 thousand tokens depending on the language.
• It has structure: article, paragraph, point. We can compare "cutting by size" with "cutting along the structure".
• It has official versions in English and Portuguese, with the same article in the same place. So the same experiment runs in both languages.

The download_gdpr.py script downloads both versions from EUR-Lex and splits the articles into a JSON file.

One detail will matter later: the EU's Portuguese version is European Portuguese, while our Portuguese questions were written the way people speak in Brazil.

The embedding model

To keep everything free, we used multilingual-e5-small: small (about 470 MB), runs on a CPU and understands both English and Portuguese.

Two of its traits shape the experiment:

• It reads at most 512 tokens. Anything beyond that is ignored, with no error and no warning. That is why the 1,000-token recipe did not even make the test: half of each chunk would be thrown away.
• It was trained with prefixes: query: before the question and passage: before the passage. Without them, search gets worse. It is the kind of detail AI-generated code tends to forget.

passages = model.encode(["passage: " + t for t in texts], normalize_embeddings=True)
queries = model.encode(["query: " + q for q in questions], normalize_embeddings=True)

Seven ways to cut

By size, ignoring the structure of the text:

• Fixed 128, 256 and 512 tokens: join the whole law and cut every N tokens.
• 500 tokens with 64 or 128 tokens of overlap: the standard recipe. Each chunk repeats the end of the previous one, so nothing is lost at the edges.

Along the structure of the document:

• By article: one chunk per article.
• By paragraph: one chunk per numbered paragraph, starting with the article's title.
• By point: like the previous one, but a paragraph that is too long becomes one chunk per point, (a), (b), (c)...

The article title at the start of each chunk is not decoration. A paragraph like "2. The processor shall notify the controller without undue delay..." does not say on its own that the topic is a data leak. The title "Article 33: Notification of a personal data breach to the supervisory authority" does.

The by-point strategy is the most elaborate one, and it still fits in a few lines:

def by_point(articles, lang, limit=400):
    chunks = []
    for a in articles:
        head = header(a, lang)  # "Article 33: Notification of a personal data breach..."
        for p in a["paragraphs"]:
            lines = p.split("\n")
            if n_tokens(p) <= limit or len(lines) == 1:
                chunks.append((head + "\n" + p, {a["article"]}))
            else:
                intro, points = lines[0], lines[1:]
                chunks += [(head + "\n" + intro + "\n" + pt, {a["article"]}) for pt in points]
    return chunks

Measuring without guesswork

We wrote 32 questions the way an ordinary person would ask them, without copying the law's wording. For each one, we wrote down the article that holds the answer:

{
  "pt": "Uma empresa vazou meus dados. Em quanto tempo ela precisa avisar a autoridade?",
  "en": "A company leaked my data. How long does it have to notify the authority?",
  "articles": [33]
}

With that, each strategy gets two numbers:

• Top-3 hit rate: for how many questions a passage from the right article shows up among the first 3 results. Three, because that is what we will send to the LLM.
• Tokens to the LLM: how much text those 3 passages add up to. That is what you pay for (in money or in time) on every question.

One last precaution: with size-based cuts, where the first cut falls is luck. So we ran each size four times, shifting the starting point, and show the average and the range.

Results

Cutting by point won in both languages, and it sends the least text to the LLM. The standard recipe ended up at the bottom of the table, while spending four times as much.

But the most interesting numbers are in the details.

What the table hides

The luck of the cut matters as much as the size

Look at the range for fixed 128 in Portuguese: from 47% to 62%. Nothing changed except where the first cut fell.

That swing is larger than the difference between 128, 256 and 512 tokens. In other words: with size-based cuts, comparing sizes is close to measuring noise. Test one size, once, and you can conclude almost anything.

A third of the law, invisible

With one chunk per article, 30 of the 99 articles go over 512 tokens. The model reads the beginning and ignores the rest.

All together, about 30% of the law's text never makes it into the search. No error, no warning: search simply cannot find what sits at the end of long articles.

The most extreme case is Article 4, the definitions. In the original, it is a single paragraph with 26 definitions and more than 2,700 tokens. Even when cutting by paragraph, only the first part is read, and the question "What counts as personal data?" misses. Cutting by point, each definition becomes its own chunk, and the problem goes away.

Portuguese loses 10 to 20 points

In every strategy, English finds more answers. Looking at the questions that missed, the reason is almost always vocabulary:

The questions are in Brazilian Portuguese and plain language. The law is in European Portuguese and legalese. In English, the distance between "leaked my data" and "personal data breach" is shorter.

This applies to any RAG over documents that are not in English: if your users speak differently from the document, search suffers, and changing the chunk size will not fix it.

Closing the loop: pgvector and a local LLM

With the strategy chosen, the rest is the usual RAG. The repository's docker-compose.yml starts PostgreSQL with pgvector and Ollama, which runs the LLM locally:

docker compose up -d
docker compose exec ollama ollama pull qwen2.5:3b
python index.py

index.py cuts the law by point and stores each chunk with its vector. Search is a SQL query: the <=> operator computes the cosine distance.

def search(lang: str, question: str, k: int = 3):
    vector = model.encode("query: " + question, normalize_embeddings=True)
    with psycopg.connect(DSN) as conn:
        register_vector(conn)
        return conn.execute(
            """SELECT article, content, 1 - (embedding <=> %s) AS similarity
               FROM chunks WHERE lang = %s
               ORDER BY embedding <=> %s LIMIT %s""",
            (vector, lang, vector, k),
        ).fetchall()

The 3 passages go to the LLM with a simple instruction: answer only from them, cite the article and say so when the answer is not there. Asking the question from the start of this article:

$ python ask.py en "A company leaked my data. How long does it have to notify the authority?"
[0.917] art. 33: 1. In the case of a personal data breach, the controller shall without undue delay...
[0.894] art. 33: 2. The processor shall notify the controller without undue delay after becoming aware...
[0.889] art. 33: 4. Where, and in so far as, it is not possible to provide the information at the same...
 
According to Article 33 of the GDPR, the controller shall notify the supervisory authority without
undue delay and, where feasible, not later than 72 hours after having become aware of the personal
data breach. Therefore, the company has 72 hours to notify the supervisory authority after becoming
aware of the data breach.

Right answer, right article. Now one of the questions that search got wrong:

$ python ask.py en "My account has wrong information. Can I demand that they fix it?"
[0.839] art. 32: 2. In assessing the appropriate level of security account shall be taken in particular...
[0.833] art. 33: 4. Where, and in so far as, it is not possible to provide the information at the same...
[0.830] art. 9: 4. Member States may maintain or introduce further conditions, including limitations...
 
Could not find.

The answer is in Article 16, which talks about the "rectification of inaccurate personal data". Search latched on to the word "account", as in "account shall be taken", and never got there. The LLM did the right thing, admitting it could not find the answer instead of making one up. But for the person asking, the RAG failed. It is the search, not the model, that sets the ceiling for the answer.

Before you generalise

32 questions is not much. Each one is worth about 3 points. Differences of 5 points may be luck; differences of 25 points, like the one between the standard recipe and cutting by point, hardly are.

The result holds for this document. A law has explicit structure. A scanned contract, a meeting transcript or a forum thread does not, and there a size-based cut may be the only option.

We were generous with size-based cuts. A 512-token chunk that catches the end of one article and the start of the next counts as a hit for both. They lost anyway.

Good search does not guarantee a good answer. We measured whether the right passage reaches the LLM. What it does with that passage is another experiment.

How to repeat this on your own documents

The takeaway is not "cut by point". It is the method:

1. Write 30 to 50 real questions, the way your users would ask them, and note where each answer lives.
2. Measure search before touching the LLM. Top-K hit rate and tokens sent already tell most of the story.
3. Test your document's structure against size-based cuts. Sections, clauses, FAQ entries: all of them are natural boundaries.
4. Check your embedding model's limits (maximum length and prefixes) before choosing a chunk size.

Writing the questions is the tedious part, and it is where an AI assistant genuinely helps. A starting point:

I will give you a document. Write 40 questions an ordinary user would ask about it,
in everyday words rather than the document's own terms. For each question, point to
the section that contains the answer. Include hard questions: ones that use synonyms,
ones that depend on a definition, and ones the document does not answer.

Review the questions and the answers before using them. They are your answer key: if it is wrong, the whole experiment measures the wrong thing.

What's next

This article is part 1 of a series on RAG.

A law is a well-behaved document: plain text with a clear structure. In part 2, we move to the real world: PDFs with tables and images, and what gets lost on the way to text.

Then, in part 3, this database becomes a tool for an agent, through MCP, continuing the series on LangGraph with FastAPI.