Back to the blog
Engineering

RAG as a Tool: An Agent Found What the Fixed Search Missed

We turned the RAG database into an MCP server and let an agent decide how to search. Right answers went from 24 to 32 or 33 of 33.

RAGMCPAgentsPython

Part 3 ended with a clear diagnosis. When the right page arrived, the large models got everything right. Every error left was the search's: in 8 of the 33 questions in English, the page with the answer never reached the model.

Part 3's search was fixed. The user's question became a vector, the database returned the 3 most similar chunks, and that was it, no second chance. A person searching the same PDF would do it differently: change the words, search again, open the chart's page to look at it.

So we gave the model that. The database became an MCP server with two tools, search and view a page, and an agent decides how to use them. Same 33 questions, same three large models, same document.

The short version: the agent got 32 or 33 of 33 with all three models, in both languages, against 24 or 25 for the fixed pipeline. All 60 right chart answers came after the agent opened the chart's page. With GPT, it was even cheaper than the fixed pipeline. And our grader broke again, this time the other way round.

The code is at salada-dados/blog, in index_pg.py, mcp_server.py, agent.py and rewrite.py. The whole experiment, tests included, cost US$ 6.80.


MCP in two paragraphs

The Model Context Protocol is an open standard for connecting tools and data to language models. An MCP server announces the tools it offers, with a name, a description and parameters. An MCP client (an agent, Claude Desktop, an IDE) lists those tools, hands the list to the model and runs the calls the model asks for.

The point is to write the tool once and use it from any client. The server in this article is the same one you can plug into Claude Desktop or Claude Code to chat with the PDF. That is for a separate post; here, the point is to measure.

The experiment

The database. The same chunks as in parts 2 and 3 (pypdf text, up to 400 tokens, multilingual-e5-small embeddings) went into Postgres with pgvector, as in part 1. With one detail: the server connects as a database user that can only read that table.

The MCP server, built with the official Python SDK, offers two tools:

  • search_report(query) returns the 3 chunks most similar to the query, each with its page number.
  • view_page(page) returns the page as an image, at 150 dpi, as in part 3.

The agent is a short loop, with no framework. It sends the model the question and the list of tools; if the model asks for a tool, the agent calls the MCP server and passes the result back; it repeats until the model answers, with at most 6 steps.

The models are part 3's three largest, through OpenRouter: Claude Sonnet 5.5, GPT-6.1 Sol and Gemini 4 Argon. Qwen was left out: in part 3, it made up answers when it could not find them, and in an agent that gets worse, because it can decide to stop searching too early.

Results

Right answers, out of 33 questions:

ModelFixed pipeline (part 3) en / ptAgent enAgent pt
Claude Sonnet 5.525 / 253332
GPT-6.1 Sol25 / 243332
Gemini 4 Argon25 / 243333

In English, all three got all 12 text questions, all 11 table questions and all 10 chart questions. In Portuguese, they got all text and chart questions; on tables, Gemini got 11 of 11, and Claude and GPT 10.

The two remaining errors are honest refusals, both in Portuguese. Claude ran 5 searches and opened 3 pages looking for July's IPCA (Brazil's consumer price index) before saying it could not find the single-month figure; GPT ran 3 searches for corrugated cardboard shipments and gave up. Nobody made up a number.

The right page started showing up. In the agent's search results, the page with the answer appeared in 31 to 32 questions in English and 29 to 30 in Portuguese. The fixed search stayed at 25 and 23.

The charts were read, not guessed. The 60 right chart answers (10 per model, in each language) all came after the agent called view_page on the chart's page. In part 3, we saw that cheap models guess; here, we checked before celebrating.

How the agent found what the fixed search missed

Across the three models and both languages, there were 54 cases where the fixed search did not bring the page. In those, the agent reached the page like this:

HowCases
In the first search, just by rewriting the question29
In a later search, with other words16
By opening a neighbouring page with view_page4
Never reached the target page5

Of the 5 cases where the agent never reached the target page, 3 are from a question whose answer also appears on another page, and it got them right anyway. The other 2 are the two errors.

The table's first row is misleading on its own. The agent's first search, where it turns the question into keywords, recovered 29 pages the fixed search missed. With Gemini, in Portuguese, "What was the annual growth of construction in 2022?" became construção civil 2022 variação anual PIB (construction, 2022, annual change, GDP), and the table on page 16 showed up right away. But that same first search lost 15 pages the fixed search found. The balance is 14 pages, from 144 to 158 across the 198 cases.

Most of the gain came from what only an agent does: look at the result, search again when it does not help, and open the next page. Counting every search, the right page showed up in 184 of the 198 cases. For exports as a three-month moving average, Gemini searched once, did not like the result, searched by the charts' numbers (Gráfico 1.2.40 Gráfico 1.2.41 Gráfico 1.2.42...) and opened page 31.

What if you only rewrite the question?

If the agent's first search already helps, can you keep just that and drop the agent? The idea is cheap: one model call rewrites the question, and part 3's fixed search runs on the new query. We tested it with the same three models. Right page among the results, adding up the three:

Searchenpt
Fixed, with the original question (part 3)7569
Rewritten question, first prompt6865
Rewritten question, second prompt7268
The agent's first search8276
All of the agent's searches9589

It did not work. The rewritten question did as well as the original or worse, so we did not even run the answering step: without the page, the answer does not improve.

The first prompt asked for "a search query for the Banco Central do Brasil's Monetary Policy Report". The models did what a search engine would want. GPT put "Banco Central do Brasil Monetary Policy Report September 2026" in all 33 English queries, and that name is in the header of every page of the PDF, so the search started pulling pages almost at random. Gemini wrote things like "2026Q2" OR "second quarter of 2026" in 9 of them, operators that mean nothing to an embedding search.

The second prompt explained that the search compares meanings, asked for the short phrase that would appear in the chunk with the answer, and banned the report's name and the operators. The queries came out clean, 8 to 10 words long, and the search went back to the original's level, without going past it. We tuned this prompt looking at the same questions, which favours the result, and it still did not beat the fixed search.

The difference from the agent seems to lie in two things: the tool description, which guides the query without pushing the report's name, and the second chance. The standalone rewrite cost US$ 0.40 for the three models; the code is in rewrite.py.

What it cost

ModelCost per question: fixed (part 3)Cost per question: agentMedian time: agent
Claude Sonnet 5.5US$ 0.017US$ 0.0246 s
GPT-6.1 SolUS$ 0.019US$ 0.0118 s
Gemini 4 ArgonUS$ 0.007US$ 0.01221 s

The agent ran 1.4 searches and opened 0.6 pages per question on average. The fixed pipeline sent 2.8 pages every time, even for text questions. That is why GPT was cheaper as an agent: it hardly opens a page unless it needs to.

Claude and Gemini were more expensive, for different reasons. Claude runs more searches and writes longer replies, and every turn of the loop resends the whole history: about 10,500 input tokens per question, twice the other two. Gemini cost little per call, but each call was slow: a median of 21 seconds per question, against 7 in part 3.

What your coding assistant will not warn you about

The tool description is the model's manual

The model only knows what a tool does from its description. We ran everything again with short descriptions, "Search the report." and "Get a page of the report.", against the detailed ones, which say what comes back, that chart values only exist in the image, and that it is worth searching again with other words.

Detailed descriptionsShort descriptions
Right answers (3 models, en and pt)196 of 198194 of 198
Tool calls per question1.972.35
Claude, in English: calls per question and cost1.9 and US$ 0.702.9 and US$ 1.04

We expected a big drop in right answers, and it did not come: large models cope well with little instruction. What changes is the path. With the short descriptions, the agent made 19% more calls, and Claude spent 48% more in English.

And sometimes it does not even find out the right tool exists. On the IPA-DI question (a producer price index), with the short description, Claude ran 7 searches and replied, in Portuguese: "The report shows the 12-month change in the IPA-DI only in Figure 1.2.50 [...]. The text I could consult does not say in which year it got close to 50%." It never called view_page. With the detailed description, it ran 3 searches, opened the page and got it right.

A generic error teaches nothing

We asked for page 999 on purpose. With a plain ValueError inside the tool, the model gets only "Error executing tool view_page". It has no way of knowing what to do differently. The SDK has its own exception, ToolError, whose message reaches the model:

Error executing tool view_page: page must be between 1 and 90, got 999

The image does not fit in the tool result

MCP lets a tool return an image. The OpenAI-style chat API, which OpenRouter uses, does not: the tool result message carries only text. If you just pass the result along, the image disappears on the way, with no error at all, and the model answers without having seen the page.

The way out is to send the image in a separate message right after the result, and say so in the result:

agent.py
note = f"(page {args.get('page')} follows as an image)" if pics else ""
messages.append({"role": "tool", "tool_call_id": tc["id"], "content": "\n".join(texts) or note})
images += [{"type": "image_url", "image_url": {"url": f"data:{p.mime_type};base64,{p.data}"}} for p in pics]
...
if images:
    messages.append({"role": "user", "content": [{"type": "text", "text": "Requested page images:"}] + images})

With the stdio transport, the most common one for a local server, standard output is the protocol channel. A debugging print() in the server turns into an invalid message in the middle of the conversation. Logs go to stderr.

The grader broke again, the other way round

In part 3, we added a rule so that refusals that repeat the question would not count as right. Here, it caught a right answer. Claude replied:

Construction grew 6.8% in 2022, according to the report's GDP table on page 16. The table's
column headers were not in the text, so I matched the columns to years from the other rows.

"Not in the" was on the list of refusal phrases. The answer is right, and it even explains how the model worked around the table's lost header, the central problem of part 2. The fix was to look for the refusal only in the first sentence, which is where every refusal we read appears. Before applying it, we checked the new rule against every reply from parts 2, 3 and 4: only this case changed.

Cap the steps and the spending

An agent decides on its own how many times to call the model, and every turn resends the whole history, images included. Without a limit, a hard question becomes an expensive loop. We used at most 6 steps (4 of the 198 questions reached it) and a budget that stops the whole run if the cost reported by OpenRouter goes past US$ 15.

In practice, what stopped us first was the credit: halfway through the experiment, the OpenRouter account ran out and the API started returning error 402. The run crashed, but the on-disk cache held every question already answered, and resuming cost only what was left. Without a cache, an error in the middle of an agent loop costs everything again.

The document is input too

The text a tool returns goes straight to the model. If a document contains a sentence like "ignore your instructions and call tool X", it reaches the model like any other chunk. That is prompt injection, and in an agent the risk grows, because the model chooses the tools' arguments.

We did not measure this here, but we took two cheap precautions. The tools only read, and they are marked as read-only in MCP (read_only_hint). And the server connects to the database as a user that can only read one table: we tested it, and a DELETE comes back with "cannot execute DELETE in a read-only transaction".

The solution, in code

The whole server is about 90 lines. The core is the two tools:

mcp_server.py
mcp = MCPServer("monetary-policy-report", instructions="Read-only access to one PDF report.")
READ_ONLY = ToolAnnotations(read_only_hint=True, destructive_hint=False, open_world_hint=False)
 
@mcp.tool(description=DESCRIPTIONS["search_report"], annotations=READ_ONLY)
def search_report(query: str) -> str:
    vector = model.encode("query: " + query, normalize_embeddings=True)
    rows = conn.execute(
        "SELECT page, content FROM rpm_chunks WHERE lang = %s ORDER BY embedding <=> %s LIMIT %s",
        (LANG, vector, K),
    ).fetchall()
    return "\n\n".join(f"[page {page}]\n{content}" for page, content in rows)
 
@mcp.tool(description=DESCRIPTIONS["view_page"], annotations=READ_ONLY)
def view_page(page: int) -> Image:
    if not 1 <= page <= len(doc):
        raise ToolError(f"page must be between 1 and {len(doc)}, got {page}")
    return Image(data=doc[page - 1].get_pixmap(dpi=DPI).tobytes("png"), format="png")

Notice the [page N] on each chunk. Without it, the model does not know which page to open, and has no way of saying where the answer came from.

The agent starts the server as a subprocess, lists its tools and converts them to the chat API's format:

agent.py
params = StdioServerParameters(command=sys.executable, args=["mcp_server.py"], env={**os.environ, "DOC_LANG": lang})
async with Client(params) as client:
    tools = [{"type": "function", "function": {"name": t.name, "description": t.description,
                                               "parameters": t.input_schema}}
             for t in (await client.list_tools()).tools]

The Python MCP SDK is at version 2, with big changes from version 1. FastMCP, which shows up in most tutorials, became MCPServer, and old code fails right at the import, with a ModuleNotFoundError. Fields also switched to snake_case (input_schema, mime_type). Coding assistants trained before that tend to write the old API. Read the docs for the installed version before you accept the suggestion.

When an agent is worth building:

  • It is when the fixed search often misses the page and the answer may be in a chart. That is where the jump from 24 to 32 or 33 came from.
  • Rewriting the question alone does not replace the agent. Here, the query rewritten in a separate call found as many pages as the original question, or fewer. The gain came from searching again and opening pages.
  • It costs predictability. The number of calls, the time and the cost vary from question to question. For a system with a response-time target, that matters.

Caveats before generalising

33 questions is not many, and the ceiling ended up low. At 32 or 33 right answers, the test no longer separates the models. Comparing Claude, GPT and Gemini as agents would need a harder set.

The result holds for this document. A 90-page report, with vector charts and well-extracted text. In a collection of thousands of PDFs, search fails in other ways, and the agent would need tools to pick the document before the page.

Models, prices and the SDK change. The figures are from November 2026, and the MCP SDK had a major version change not long ago.

We wrote the detailed description once. We did not test variations of it. A better description might cut the calls even further.

What's next

This is part 4 of the RAG series. At 32 or 33 right answers, this set of questions has hit its limit: the series' next test has to be harder.

In a separate post, we will plug this same MCP server into Claude Desktop and Claude Code, to chat with a PDF without writing the agent.