RAG over PDFs: What Gets Lost Between the Table and the Text

In part 1, the document behaved: a law, plain text, numbered articles. In real life, the document that arrives is a PDF. With tables, charts, two columns and footnotes.

Ask an AI assistant for a RAG over PDFs and the code is almost always the same: pypdf to extract the text, a text splitter, embeddings, done. Maybe it suggests a library "that understands tables". It rarely asks what happens to the number that was in the table.

So we measured it. We took the Banco Central do Brasil's Monetary Policy Report, wrote 33 questions with known answers (in the text, in a table, or only in a chart) and compared four ways of turning the PDF into text, in English and in Portuguese.

The short version: the most sophisticated extractor rebuilt the tables far better than the others, and the LLM still read the wrong cell. Charts only started getting answers once a vision model described them, at a cost of two hours per language. Along the way we found half a dozen traps that AI-generated code does not handle. At the end, we show how to read tables and charts properly with a large model over an API, and what that costs.

The full code, with the questions, is at salada-dados/blog. Everything runs for free, on your own machine, with no API key.

Why PDF is hard

A PDF does not store text. It stores drawing instructions: "write '2.8' at position (512, 340)". There is no "row 3, column 11" in a table, and no "this is a chart". Whatever extracts the text has to guess the structure from the positions.

Here is the same row of Table 1.2.1 (GDP by sector) coming out of three extractors. The question is: how much did agriculture grow in the second quarter of 2026, quarter on quarter? The answer is 2.8.

pypdf returns the text in the order it was drawn:

 Itemization  YoY change
 2021  2022  2023  2024  2025
 Q1  Q2  Q3  Q4  Q1  Q2
 GDP at market prices 4.8 3.0 3.2 3.4 2.3 1.3 0.4 0.1 0.2 1.1 0.5
    Agriculture 0.0 -1.1 16.3 -3.7 11.7 16.1 -3.2 0.5 0.7 2.3 2.8

The numbers are all there. What is gone is the header: "Q1 Q2 Q3 Q4 Q1 Q2" does not say that the first four quarters are from 2025 and the last two from 2026. To find the 2.8, you have to count columns.

pymupdf4llm tries to build a Markdown table, and the header comes out chopped up:

|Itemization|YoY ch|ange||||||QoQ,|s.a.||%|
||2021|2022|2023|2024|2025||20|25||20|26|
|||||||Q1|Q2|Q3|Q4|Q1|Q2|
|Agriculture|0.0|-1.1|16.3|-3.7|11.7|16.1|-3.2|0.5|0.7|2.3|2.8|

IBM's Docling uses a layout model to rebuild the table:

| Itemization | YoY change | YoY change | YoY change | YoY change | YoY change |      |      | QoQ, | s.a. |    | %    |
|             | 2021       | 2022       | 2023       | 2024       | 2025       | 2025 | 2025 | 2025 |      |    | 2026 |
|             |            |            |            |            |            | Q1   | Q2   | Q3   | Q4   | Q1 | Q2   |
| Agriculture | 0.0        | -1.1       | 16.3       | -3.7       | 11.7       | 16.1 | -3.2 | 0.5  | 0.7  | 2.3 | 2.8  |

It is not perfect: the quarterly group title still comes out in pieces, and the year row is only half right (2025 sits over Q1 to Q3, Q4 has no year, and 2026 appears only over Q2). But the words are whole, "YoY change" is repeated over every column it covers, and every number sits in its column. It is by far the most readable version.

The question of the experiment is whether that difference, so obvious to the eye, makes it all the way to the answer.

The document: the Central Bank's Monetary Policy Report

The Monetary Policy Report is the quarterly document in which Brazil's monetary policy committee (Copom) explains the economic outlook and its inflation projections. We used the September 2026 edition, for three reasons:

• It has everything: about 90 pages, 20 tables and 100 charts, two-column text, boxes and footnotes.
• It has official versions in English and in Portuguese, with the same tables and charts. The same experiment runs in both languages.
• It can be reproduced if the source is cited: "Monetary Policy Report, volume 2, nr. 3".

Before writing any code, we opened the PDF and looked. That avoided the first mistake. The English version has 317 images, and the Portuguese one, two. It looked as if one language had charts as images and the other did not. It was the cover, cut into hundreds of pieces. In both languages, the charts are vector graphics: lines and text that are drawn, not pictures.

That changes everything for charts: axis labels, legend and title are text, and a simple extractor reads them.

Four ways to extract

All on CPU, on an ordinary laptop. The vision model is qwen2.5vl:3b, running on Ollama.

From the extractor onwards, nothing changes between the four: the same text splitter (blocks of up to 400 tokens, never crossing a page), the same embedding model as part 1 (multilingual-e5-small), the same 3 passages sent to the LLM, and the same LLM (qwen2.5:3b). If the result changes, it was the extractor.

Measuring without guesswork

We wrote 33 questions, each with its page and exact answer:

• 12 text questions: the answer is in a paragraph. "How much did GDP grow year over year in the second quarter of 2026?"
• 11 table questions: the answer is in a cell. "How much did Brazil import from January to July 2024?"
• 10 chart questions: the answer only exists in the drawing. "In which year did the unemployment rate reach its highest value since 2004?" We picked questions whose answer is not written anywhere in the report's text.

And we measured three things, in sequence:

1. Right page: among the 3 passages, did the search bring one from the page that holds the answer?
2. Answer in context: is the exact answer (the 2.8, the "Japan") written in those passages?
3. Right answer: did the LLM answer with it?

Part 1 stopped at the first number. That is not enough here: with tables, the search can hit the right page and the number can still arrive without meaning.

Grading is automatic, and one detail matters: "2.8" must not count as a hit when the reply says "12.8" or "2.85". So the answer has to appear as a whole number, not as a piece of another one:

def contains(text: str, answers: list[str]) -> bool:
    for a in answers:
        if re.search(rf"(?<![\w,.]){re.escape(a)}(?![\d]|[,.]\d)", text, re.I):
            return True
    return False

Finally, a no-RAG baseline: the same questions to the LLM, with no passages at all. If it got them right on its own, a hit with RAG would prove nothing. It got 0 of 33 in English and 1 in Portuguese, and that one hit was "urea or sulphur?", which is a coin toss.

Results

Right answers, out of 33 questions:

The text, table and chart columns are from the Portuguese run, where the differences are clearer. English was weaker across the board, and we come back to that below. The tokens sent to the LLM stayed between 830 and 1,080 per question for every extractor, so no one won by sending more text.

At first sight, the story is simple: the more sophisticated the extractor, the better, and the vision model is what unlocks the charts. But look at the table column: 2 of 11 for everyone. The extractor that rebuilt the table scored the same as the one that scrambled the header.

What the table hides

The small LLM reads the wrong cell

We read the replies one by one. On table questions, the problem was almost never a missing number. It was the LLM picking the neighbour. Some examples from the Portuguese run (the English run made the same kind of mistakes, including the 2028 row in the last one):

qwen2.5:3b has 3 billion parameters. Following a row to the right column in a 12-column table is hard for it, with or without structure.

To test that hypothesis, we ran the table questions again with qwen2.5:7b, on exactly the same passages. Counting only the questions where the right number reached the LLM (English and Portuguese together):

With the small model, structure makes no difference. With the larger one, it does. With pypdf, without the header, even the 7B gets lost: asked about light vehicle production in August (in Portuguese), it answered "-0.0", the value in the next column.

In other words: rebuilding the table only pays off if the LLM can read a table. Paying 15 to 30 times more extraction time and handing the result to a 3B model is throwing the work away.

Half of the tables never arrive

Even with the 7B, the other problem remains: in about half of the table questions, the number was not in the 3 passages. The search did not bring the table.

The clearest case is pymupdf4llm: it found the right page in 9 of 11 table questions, but the number was in the context in only 3. The search brought the paragraph that talks about the table ("GDP grew 0.5%... agriculture grew strongly"), not the table itself.

That makes sense. The question "how much did agriculture grow" looks a lot like a sentence about agriculture. A block of |0.0|-1.1|16.3|-3.7| looks like nothing. And Markdown tables are expensive: the page with Table 1.2.1 costs about 1,000 tokens as plain text and 1,700 to 1,850 with the tables in Markdown. With a 400-token limit per chunk, the table is sliced into several chunks, and only the first one carries the header.

That points to the next step, left for another experiment: store each table with a text summary ("Table 1.2.1: GDP by sector, yearly change from 2021 to 2025 and quarterly up to 2026Q2") and repeat the header in every chunk.

Docling throws away the text in the charts

In the chart column, Docling without vision was the worst at finding the right page: 4 of 10 in English and 5 of 10 in Portuguese, against 9 and 7 for pypdf.

The reason: when it recognises a chart as a figure, Docling replaces everything inside it with <!-- image -->. The title, the legend and the axis labels, which pypdf dumps as loose text, disappear. And Docling almost never links the figure to its caption: 88 of 126 figures were left without a caption in English, and 96 of 123 in Portuguese. In several cases, two side-by-side charts became a single figure.

A description without "Figure 1.2.14 – Unemployment rate" is much harder to find. So we read the caption straight from the PDF, in the area just above the figure:

def caption(item, doc, pdf):
    if text := item.caption_text(doc).strip():  # when Docling did link the caption
        return text
    prov = item.prov[0]
    page = pdf[prov.page_no - 1]
    box = prov.bbox.to_top_left_origin(page_height=page.rect.height)
    area = pymupdf.Rect(box.l, box.t - 30, box.r, box.b)  # the figure and a little above it
    ...  # the first line starting with "Figure 1.2.14 –" is the caption

With that, 118 of the 123 figures in Portuguese got their caption. The ones left are the cover and icons.

The vision model describes before the question

With captions sorted, each figure was described by qwen2.5vl:3b, and the description replaced the <!-- image -->. That is what took charts from 0 to 4 right answers in Portuguese.

The first prompt, though, was a disaster. It was detailed: title, axes, unit, every series, the maximum and minimum of each. The model made up a title that does not exist ("Crescimento Econômico: Projeções 2025-2026"), got stuck repeating "Não especificado no gráfico" (not specified in the chart) for 32 combinations of country and quarter, and took 7 and a half minutes on a single chart. Across 250 charts, that would be 31 hours.

A short prompt, with an output limit, fixed it:

PROMPT = ("Describe this chart in up to 5 sentences, in English: what it measures, the period, "
          "the series in the legend and what stands out (peaks, lows, which is higher or lower, "
          "approximate values). Use only what is visible.")
 
options = {"temperature": 0, "num_predict": 300, "repeat_penalty": 1.15}

About 66 seconds per chart. But look at what it wrote about Figure 1.1.4, GDP growth by country:

This bar chart compares economic data across different regions for various quarters from 2025Q3 to 2026Q2.
...
The Global region shows a peak around 5 by 2026Q2 compared to other quarters. The UK also exhibits
a notable rise from its initial value of about 1 in 2025Q3 to nearly double it by the end of 2026Q2,
indicating strong growth.

"Economic data". The question was which country grew the least, and the description does not say. Worse: it states, with full confidence, that the UK "nearly doubled" and that the global figure peaked around 5. Neither is in the chart. The Portuguese description of the same chart did not invent numbers, but it was just as vague ("um indicador específico", a specific indicator). That may be why vision barely helped in English (1 of 10).

The underlying problem is the order of things. The model describes each chart once, before any question exists, without knowing what will matter. Afterwards, only the description goes to the search, and neither the search nor the LLM ever sees the image.

So why, when you send a screenshot to an assistant like Claude or ChatGPT, does it read everything correctly? Because there it is the other way round: a much larger model looks at the image together with your question. That is where the way out is.

So how do you read them properly?

Everything so far was free, on CPU, with 3 and 7 billion parameter models. That was on purpose: you can repeat it without spending anything, and the flaws show up clearly. But if the goal is to get tables and charts right, the way to go is a large model, over an API. It has a cost and it has risks. And it is the solution.

What the experiment already tells us

The numbers above separate the problems well, and each one has a different remedy:

• Reading the table is a matter of model size. On the same passages, the 7B went from 5 to 10 right answers out of 12 on Docling tables. A frontier model reads a structured table without effort.
• Reading the chart is a matter of order. The model has to look at the image with the question in hand, not describe it beforehand, blindly.
• Finding the table does not improve with any model. If the search did not bring the passage, the most expensive LLM in the world has nothing to read. That is fixed in the index (a table summary and a repeated header), not in the LLM.

The architecture: search the text, answer from the image

The idea is to do in the pipeline what the chat does:

1. Search stays on text. Extract with pypdf or Docling, as we did. The search is only there to find the page.
2. The answer comes from the page image. Render the pages of the 3 passages found and send the images, together with the question, to a large model with vision.

import base64, pymupdf
 
def page_png(pdf, page_no, dpi=150):
    pix = pdf[page_no - 1].get_pixmap(dpi=dpi)
    return base64.b64encode(pix.tobytes("png")).decode()
 
pages = sorted({page for _, page in top3})        # pages of the 3 passages found
content = [{"type": "image", "source": {"type": "base64", "media_type": "image/png",
            "data": page_png(pdf, p)}} for p in pages]
content.append({"type": "text", "text": question})
# client.messages.create(model=..., messages=[{"role": "user", "content": content}])

Notice that here the "bad extractor" helps. pypdf found the right page in 9 of 10 chart questions in English (7 in Portuguese), precisely because it dumps the chart's labels and legend as text. With the page image going to the model, that is the ceiling: at most 9 of 10 in English, because on the remaining question the page never arrives. Even so, it is far above the 1 of 10 we got in English with two hours of local model.

The simpler alternative is to keep the pipeline we already tested and only swap the model that describes the figures for a large one. The descriptions get better, but it still describes before the question.

What it costs

The report's pages are A4. Rendered at 150 dpi, each one is 1241 × 1754 pixels. Using Anthropic's formula (pixels ÷ 750), that is about 2,900 tokens per page; other providers use similar formulas. At mid-tier model prices of US$2 per million input tokens and US$10 per million output tokens (November 2026 prices; check before budgeting):

Two cents per question looks like nothing, and for a test it is. In production, at 1,000 questions a day, that is around US$600 a month, just for images. A top-tier model costs 2 to 4 times more. You can cut it: send only the page of the first passage, lower the resolution, send the image only when the question is about a number, or use batch processing when there is no hurry.

These figures are estimates, not measurements. The test with the large model is the next article.

The risks

• The document leaves your machine. A public Central Bank report, no problem. Contracts, medical records, customer data: then data protection law (GDPR, LGPD), your contract with the provider (retention, use for training, region where it runs) and sometimes banking or professional secrecy come in. Ask legal before, not after.
• Cost scales with use, not with the document. With the local model, the cost was the time to index. Here, every question pays again. Put a spending cap in place and measure cost per question from day one.
• Large models get things wrong too. Reading a value in the middle of a curve is an approximation, for people and for models. Ask for the page the answer came from and keep the strict grading we used here.
• The image is input too. Text hidden in a figure ("ignore the previous instructions") reaches the model like any other text. With third-party documents, that is a prompt injection vector.
• Vendor dependence. Prices, models and usage limits change. Keep the questions and the grader from this experiment: they are what tells you whether switching models made something worse.

If none of this can leave the building, there is a middle ground: a larger vision model running locally, on a GPU. It costs more in hardware, but the document stays home.

What your coding assistant won't tell you

Putting it all together, here is what made a difference in this experiment and that rarely shows up in code generated by an AI assistant:

1. Open the PDF before choosing the tool. The 317 images were the cover; the charts were vector graphics. The choice of extractor depends on that.
2. Check the link between figure and caption. Docling left 70% to 78% of the figures without a caption, with no error and no warning.
3. Test the vision prompt on one figure before running a batch. The "complete" prompt would have cost 31 hours and produced made-up descriptions.
4. Measure the answer, not just the search. Right page is not right number, and right number is not right answer.
5. Run a no-RAG baseline. Without it, you do not know how much of the score is the document and how much is guessing.
6. Grade strictly. An if "2.8" in reply accepts "12.8".
7. Read the licences and the side effects. pymupdf4llm depends on PyMuPDF, which is AGPL (or a commercial licence), and that matters if your code is closed source. It also runs OCR on its own on pages without text (here, the cover), without anyone asking.

Before you generalise

33 questions is not many. There are 10 to 12 per type, so each question is worth almost 10 points. That is why we show counts, not percentages. Differences of one or two hits may be luck; the 7B's "10 of 12 against 7 of 11" is strong evidence, not proof.

The result holds for this document. The report is a born-digital PDF with vector charts. In a scanned PDF, pypdf extracts nothing and OCR comes into play. With charts as images, the labels that rescued pypdf here would not exist.

The models are small on purpose. Everything runs for free, on CPU. The results measure the extractors and show where a small model breaks. They are not the ceiling of what a large model can reach.

English did worse on almost everything, and we did not dig deep into why. The questions are translations of the same questions, and the document is the same. The made-up description of Figure 1.1.4 is a clue, not the whole explanation.

How to repeat this on your own PDF

1. Look at the PDF. Born-digital or scanned? Vector charts or images? How many tables? pymupdf answers that in five lines.
2. Write questions separated by type: text, table, figure. They fail for different reasons, and the average hides that.
3. Measure the three steps: right page, answer in context, right answer. That is how you find out whether the problem is extraction, search or the LLM.
4. Only pay for the expensive extractor if the rest of the chain makes use of it. A rebuilt table with a small LLM gained nothing here.
5. Test the vision model on three or four figures and read the descriptions before running it on the whole document.

What's next

This is part 2 of the RAG series. The next article puts the architecture above to the test: the same 33 questions, with the pages going as images to a large model, and the real cost measured. It also explains why the chat reads your screenshot so well and the pipeline does not.

After that, in part 3, the database becomes a tool for an agent, via MCP, continuing the LangGraph with FastAPI series.