Back to the blog
Engineering

Why the Chat Reads Your Screenshot and Your RAG Does Not

We sent the PDF page as an image to four large models. Charts went from 0 to as many as 9 of 10, for less than 2 US cents per question.

RAGPDFLLMPython

You take a screenshot of a chart, paste it into Claude or ChatGPT and ask which country grew the least. It gets it right, sometimes with the approximate value too. Then you put the PDF of that same report into a RAG, ask the same question and get "I could not find it".

In part 2, that is what happened with the Banco Central do Brasil's Monetary Policy Report. With four PDF extractors and a local model with 3 billion parameters, charts stayed at 0 of 10. The best result, 4 of 10, cost two hours of vision model per language, with generic and even made-up descriptions.

At the end of that article, we proposed a way out: search the text and answer by looking at the image of the page, with a large model through an API. We promised to test it. We did, with four models, the same 33 questions and the cost measured call by call.

The short version: charts went from 0 to 9 of 10 in English and 7 of 10 in Portuguese, exactly the ceiling the search allows. When the right page arrived, the three largest models got every question right. Each question cost between US$ 0.007 and US$ 0.019. And along the way our grader broke twice, in a way worth telling.

The full code is at salada-dados/blog, in vision.py. This one ran with an API key: the whole experiment, tests included, cost US$ 3.94.


Why the chat reads your screenshot

The chat model is not magic. The difference is in what it receives and when.

In the chatIn the part 2 pipeline
Who looks at the imageA large modelA 3B model, on a CPU
When it looksTogether with your questionBefore any question exists
What it looks forThe answer to your question"Describe this chart in up to 5 sentences"
What reaches the LLM that answersThe imageOnly the description, as text

The second row matters most. Describing a chart blind means writing a generic summary, without knowing what will matter. In part 2, the vision model described Figure 1.1.4, GDP growth by country, as "economic data", and stated that the UK "nearly doubled", which is not in the chart. Nobody can find the country that grew the least in that description.

With the image and the question together, the model looks for one thing: the shortest bar. That is what all four models in this test did, answering "Japan" from the page.

The experiment

Everything before the answer stayed the same as in part 2, so the comparison is fair:

  • The same document, in its English and Portuguese versions, and the same 33 questions: 12 answered in the text, 11 in a table and 10 only in a chart.
  • The same search: text extracted with pypdf, chunks of up to 400 tokens, multilingual-e5-small embeddings and the 3 most similar chunks.

Only the model that answers changed. Each of the four models answered every question twice:

  • Text: the same 3 text chunks the small model got in part 2.
  • Image: the pages those chunks came from (2.8 pages per question on average), rendered at 150 dpi.

Comparing the two modes separates two things that usually come mixed: the gain from a larger model and the gain from seeing the page.

As in the other parts, there is no RAG framework in the middle, no LangChain and no LlamaIndex: each step is a short function, in plain sight. That is how we make sure only one thing changes between one test and the next.

The models, all called through OpenRouter, which gives access to several providers with a single key:

ModelProviderPrice per million tokens (input / output)
Claude Sonnet 5.5AnthropicUS$ 2 / 10
GPT-6.1 SolOpenAIUS$ 2 / 10
Gemini 4 ArgonGoogleUS$ 2 / 10
Qwen3-VL 235BAlibaba (open model)US$ 0.21 / 1.90

The first three are in the same price range. The large Qwen is from the same family as part 2's qwen2.5vl:3b, only 80 times bigger: it shows how far a cheap open model goes.

Results

Right answers, out of 33 questions:

ModelModeText (12)Table (11)Chart (10)Total (en)Total (pt)
qwen2.5:3b (part 2)text630910
Claude Sonnet 5.5text8501314
image9792525
GPT-6.1 Soltext9511515
image9792524
Gemini 4 Argontext9501414
image9792524
Qwen3-VL 235Btext9531715
image7661923
Search ceiling9792523

The columns by type are from the English run. The "search ceiling" is the number of questions where the page holding the answer was among the pages sent.

Three things stand out.

A larger model reading the same text helps little. From 9 to between 13 and 15 right answers for the three large models. Most of the gain is in tables, from 3 to 5: with the right number in the context, the large model does not read the neighbouring cell, as the 3B did. But charts stay at zero or close to it, because the number is not in the text. No model, of any size, reads what it did not receive.

Seeing the page is what changes the game. With the image, charts went to 9 of 10 in English and 7 of 10 in Portuguese. That is exactly the ceiling: in the other questions, the search did not bring the chart's page.

When the right page arrived, the three largest models got everything right. Across both languages, 48 questions had the target page among the images, and Claude, GPT and Gemini answered all 48. Every error left was the search's. The bottleneck moved: it is no longer reading, it is finding.

Qwen 235B was a step behind, with 41 of those 48. It misread numbers in charts and tables ("4.6%" instead of 4.1, "2024" instead of 2025). At a fifth of Gemini's price and a thirteenth of Claude's, that is a reasonable result, but not the same one.

What it really cost

In part 2, we estimated about 2,900 tokens per A4 page at 150 dpi and around US$ 0.02 per question. Now we have the numbers OpenRouter returns with each call:

ModelTokens per pageCost per question (image)1,000 questions a day, per month
Gemini 4 Argon~1,100US$ 0.0066~US$ 200
Claude Sonnet 5.5~2,850US$ 0.0169~US$ 510
GPT-6.1 Sol~2,570US$ 0.0185~US$ 560
Qwen3-VL 235B~2,150US$ 0.0013~US$ 40

The estimate held for Claude, whose image token formula we used in part 2. But Gemini charges less than half the tokens for the same page, which is why it ties with the other two on right answers at a third of the price. Each provider has its own way of counting image tokens, and it weighs more than the price per token.

Text mode costs US$ 0.003 to US$ 0.005 per question with the three large models. Images cost 2 to 5 times more, and that is still little next to part 2's alternative, which took two hours per language to describe the figures and got 4 of 10.

The median response time was 4 seconds per question for Claude, GPT and Qwen, and 7 seconds for Gemini.

Prices from November 2026, without the fee OpenRouter charges when you buy credits. Check them before budgeting: they change often.

What your coding assistant will not warn you about

The grader breaks when the model changes

In the first test run, with two questions, one reply counted as right. In Portuguese, it said:

Não encontrei essa informação nas páginas fornecidas, que tratam apenas da atividade
econômica do Brasil e não trazem dados de crescimento do PIB de EUA, China, Japão,
área do euro ou Reino Unido.

That is: "I could not find this information in the pages provided, which only cover Brazil's economic activity and have no GDP growth data for the US, China, Japan, the euro area or the UK." The right answer was "Japão" (Japan), and there it is, in the list Claude repeated from the question. Part 2's 3B model almost always refused with a short sentence. A large model that explains why it is refusing breaks the rule.

The fix was to detect the refusal before looking for the answer:

evaluate.py
REFUSAL = re.compile(r"não (encontrei|foi possível|está)|não (há|consta|aparece)|"
                     r"could not find|couldn't find|cannot find|can't find|"
                     r"not (found|given|shown|provided|in the)|(does|do) not provide",
                     re.I)
 
correct = contains(reply, answers) and not REFUSAL.search(reply)

It was not a one-off. In the full run, this rule caught 9 refusals that would have counted as right, 8 of them from Claude, the model that explains its refusals the most. Without it, Claude would have scored 7 more right answers in text mode alone, and the conclusion "a larger model reading text helps little" would look weaker than it is. We then looked for the same flaw in part 2 and found one: with pypdf, in English, "the excerpt does not provide a direct comparison of the rise in urea and sulfur prices" counted as right for the question "urea or sulphur?".

The list of phrases is not exhaustive, and it cannot be. It covers the refusals we read in the replies from parts 2 and 3. A generic refusal phrase would also catch right answers: in Portuguese, "não trazem" ("do not have") appears in a right answer from Claude, which read the petrol price peak off the chart, "about R$ 6.78 per litre, read from the chart, since the pages do not have the number in the text". Every new rule has to be checked against the right answers, not just the wrong ones.

The second slip went the other way. In English, the models answered "958,000" to the question on net job creation, and the answer key said "958" (thousand). The rule that keeps "2.8" from matching "12.8" also rejected "958,000". A right answer counted as wrong, and this was already happening in part 2: every extractor lost one right answer in English. We fixed both flaws, regraded part 2 from the saved replies and added a correction note to that article. Its conclusions did not change.

The lesson holds for any automatic grading: the grader was built for one model's replies. When you change the model, read the replies again, one by one, before you trust the count.

Cheap models guess

When the search did not bring the right page, the expected reply is "I could not find it". In 18 such questions (English and Portuguese), in text mode:

ModelSaid it could not find itAnswered something wrong
Claude, GPT and Gemini180
Qwen3-VL 235B612

Qwen made up an answer in two thirds of the cases.

The same habit shows in the chart hits in text mode. In questions like "which country grew the least?", the chunks had the candidates (the legend's names, the axis's years), but not the values that say which one is lowest. Claude and Gemini said there was no way to tell. Qwen picked a candidate and was right 4 times; GPT, 2. Luck or something they already knew, there is no way to say, and that is why this kind of hit is worth little.

In a RAG, "I could not find it" is a useful answer: it tells the user the system failed. A made-up number delivered with confidence is worse than none.

Image token counts are not universal

If you estimate cost with one provider's formula, as we did in part 2, you will be right for that provider and wrong for the others. Here, the same page cost between 1,100 and 2,850 tokens. Measure with two or three real calls before budgeting, and keep the cost the API returns with each reply.

Sending the page means sending the document

In image mode, the whole page leaves your machine, with everything on it, not just the chunk the search found. For a public report, that is fine. For a contract, a medical record or customer data, GDPR (or Brazil's LGPD) and your contract with the provider come into play, and going through an intermediary like OpenRouter adds one more company to that list. The risks we listed in part 2 (pay-per-use cost, prompt injection hidden in an image, vendor lock-in) still apply.

The solution, in code

The core of vision.py fits in a few lines. The search is part 2's. After it, the pages of the 3 chunks found go as images, along with the question:

vision.py
import base64, pymupdf, requests
 
def page_png(doc, page, dpi=150):
    pix = doc[page - 1].get_pixmap(dpi=dpi)
    return base64.b64encode(pix.tobytes("png")).decode()
 
pages = sorted({page for _, page in top3})  # pages of the 3 chunks found
content = [{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{page_png(doc, p)}"}}
           for p in pages]
content.append({"type": "text", "text": question})
 
r = requests.post("https://openrouter.ai/api/v1/chat/completions",
                  headers={"Authorization": f"Bearer {key}"},
                  json={"model": "google/gemini-4-argon", "temperature": 0,
                        "messages": [{"role": "system", "content": prompt},
                                     {"role": "user", "content": content}],
                        "usage": {"include": True}})  # returns the cost of the call

The key lives in an environment variable, never in the code. And the script caches every reply on disk, so rerunning to change the grader costs nothing. That is how we fixed both slips above without spending another cent.

When this architecture is worth it:

  • It is when the answer may be in a chart or a table, the document is allowed to leave your machine and the volume is moderate.
  • It pays off less for questions answered in the text: there, text mode got 8 or 9 of 12, against 9 with images, for a fraction of the price. A middle ground is to send the image only when the question asks for a number.
  • It does not fix the search. If the page does not arrive, no model gets it right. With a scanned PDF, pypdf extracts no text, and the search needs OCR or another strategy.

Caveats before generalising

33 questions is not many. Differences of one or two right answers between Claude, GPT and Gemini mean nothing. What the sample shows clearly is the gap between text and image, of about 10 right answers.

The result holds for this document. The report is a born-digital PDF with vector charts, and the labels pypdf extracts are what lets the search find a chart's page. With charts as images, that ceiling would be lower.

Models and prices change. The names and figures are from November 2026. What should last is the design of the test: the same pipeline, with text and with images, measured question by question.

One more intermediary. OpenRouter made it easy to compare four providers with a single key. In production, talking to the provider directly removes one link from the data chain, and the fee.

What's next

This is part 3 of the RAG series. The bottleneck now is search: in 8 of the 33 questions in English, the right page never reached the model. Improving that, with summaries of tables and charts in the index or with multimodal search straight over the page images, is the next experiment.

Before that, in part 4, the database becomes a tool that an agent queries through MCP.