How Agentic RAG Learns to Say "I Don't Know"

Say you've just joined a new company. It's your second week, you twist your ankle at Saturday football, spend a night in hospital, and now you want the insurance to cover the bill. There's an internal chatbot for exactly this kind of thing, so you ask it:

1How many days do I have to submit a health insurance claim after discharge?

Everyone around me seems to be building agentic RAG systems for questions like this one. But if I asked you to build one from scratch, I bet your first instinct would be: why even RAG? Wouldn't the LLM already know?

Wouldn't the model already know?

Well, LLMs are trained on the public internet, more or less anything accessible to the general public. Your company's claims policy sits on an internal wiki behind a login, so the model has never seen it.

You'd hope it would admit that. It rarely does. A model is trained to predict the next word, and the most likely words after your question are words that sound like an answer:

1You have 30 days from the date of discharge to submit your claim, along with the
2discharge summary and the itemised hospital bill.

It reads like it was copied out of the policy. Your actual policy says 15 days. (I made up both numbers for this example, which is what the model did too. The difference is that it didn't tell you.)

When a model fills a gap in its knowledge with something made up like this, we say it's hallucinating. For facts that only exist inside your company, these fabrications are more often than not false, and you only find out when your claim gets rejected.

Then just hand it the document

So if the model doesn't know about something, what can we do?

Think about how you already use these models. When you want a summary of a long article, you paste the whole thing in and ask for one. The model has never seen that article, and the summary still comes out fine, because the model is reading the article you gave it. It can work with what it learned in training and with whatever you pass it in the conversation.

So shouldn't it be possible to do the same here? Paste the claims policy into the chat, then ask the question. The model reads it, finds "15 days", and gets it right.

And this is exactly what RAG, Retrieval-Augmented Generation, is all about: find the internal piece of knowledge, pass it to the LLM along with the question, and let it answer from that.

But there's a catch. There's a limit to how much text a model can take in at once, and it's called the context window. It's measured in tokens (I wrote a whole article on tokens if you want to know what they really are). Anything beyond that limit gets rejected or cut off before the model ever sees it. That's why a long chat seems to forget what you said at the start: the app quietly drops the oldest messages to stay under the limit, and your early instructions go with them. Even well inside the limit, models are noticeably worse at using information buried in the middle of a very long prompt.

So we can't just dump the company's entire wiki into the chat and ask our question. And even if we could, it would cost us. LLM providers charge for the tokens you send as well as the tokens you get back, so passing the whole wiki with every question means paying for thousands of pages just so the model can read one paragraph.

Fetch only what the question needs

What we really want is a way to pass in only the documents the question needs. If I ask about my health insurance claim, the system should fetch the claims policy, maybe the list of network hospitals, and send just those to the model along with my question. The leave policy and the Wi-Fi setup guide can stay out of it.

That cuts both the irrelevant text and the bill. But how is anyone supposed to know which piece of information to fetch for a given question?

Okay, but which piece?

The obvious answer is keyword search, the kind Elasticsearch has been doing for years: take the words in the question and find the documents that contain them. It's fast and it works well, as long as the person asking uses the same words as the document.

People usually don't. Say you ask the bot:

1will the company pay for my hospital stay?

Keyword search goes looking for "pay", "hospital" and "stay". It finds the payroll FAQ, because it says "pay" a lot, the guest house rules, because they say "stay", and the page on network hospitals, which is at least the right topic. The page that actually answers you is called "Claiming inpatient expenses". It shares none of those words, so keyword search walks right past it.

Semantic search finds documents by meaning instead. It knows a "hospital stay" and "inpatient" treatment are nearly the same thing, so the claims page comes back even though the two don't share a single word.

How does it know? Embeddings, which I covered in the previous article. Every piece of text is turned into a list of numbers, a point in a space with hundreds or thousands of dimensions, and text with similar meaning ends up close together. When you ask something, your question goes through the same embedding model, so it lands in that same space as a point of its own. Searching then means finding the pieces of text whose points are closest to your question's point, usually measured with cosine similarity, which the embeddings article explains.

Semantic search has a blind spot of its own, though. If you search for a claim ID like CLM-48213 or a form called "Form 4B", meaning doesn't help much; you want that exact string. That's why it's always better to run both and merge the results, which is called hybrid search. Keyword search (usually BM25, the ranking Elasticsearch uses) catches the exact terms, and semantic search catches everything that says the same thing in different words.

One paragraph, forty pages

But who's stopping the search from fetching a 40-page employee handbook because one paragraph on page 14 talks about claims? Technically it's a match. It's also 40 pages sent to the model for the sake of one paragraph. And a single vector for the whole handbook is an average of the leave policy, the travel rules and the code of conduct, so it doesn't even sit particularly close to your question. (It's the same problem "bank" had in the embeddings article: one point for everything ends up being a smoothie of everything.)

To solve this, we break documents into small pieces called chunks before indexing them. There are different ways to split: on headings or paragraphs, every few hundred tokens with a little overlap so no sentence gets cut in half, or with smarter algorithms that split wherever the topic changes. Each chunk also gets tags, like the document it came from, its section, its date and who's allowed to see it, so we can filter on them later.

And where are these chunks stored? Each one goes through an embedding model, which turns it into a vector, and the vector is stored along with the chunk's text and tags in a vector database. Its job is easy to describe: given your question's vector, return the chunks whose vectors are closest, and do it fast, even when there are millions of them.

So, to sum up

We first break our documents down into small chunks, pass them through an embedder, and store their vectors in a vector database. That part happens ahead of time.

Then, when a question about health insurance comes in, we embed it with the same model, find the chunks closest to it (plus the keyword matches, if we're doing hybrid search), and pass those chunks to the LLM along with the question.

It's the same as you opening the policy, finding the paragraph about claims, and pasting it into the chat with your question. The only difference is that the vector database does the finding.

This whole thing is what we call RAG. It works surprisingly well, and plenty of systems in production are exactly this: a straight line from question to answer.

Nobody checks the work

Look at that pipeline again. Nowhere in it does anything check whether the documents it found were the right ones, or whether the answer it gives is actually correct.

The search returns the top five chunks and the pipeline passes them along, whatever they are. Maybe you asked about the claim deadline and all five chunks are about network hospitals: close in meaning, no use for the answer. Maybe one of them is from the 2021 version of the policy, back when the window was 30 days.

Then the model writes its answer, and nobody checks that the answer is supported by the chunks it was given. If none of them mention a deadline, a model trying to be helpful will fill the gap from its training data, and we're back at the start of this article. A confident guess, except now it comes with a citation that doesn't say what the answer says.

A person doing this research wouldn't stop there. You'd skim what you found, notice it's all about hospitals and nothing about deadlines, and search again with better words.

Aligned, but what the hell is agentic RAG?

Agentic RAG wraps the model in a loop so it can work the way that person would. People often use "agent" and "LLM" as if they were the same thing, so let's separate them. The agent is ordinary code: it holds the question, the chunks found so far and how many searches it has used, and it's the one that actually runs the search. The LLM is something the agent calls whenever it needs a judgment. The agent asks, the LLM decides, the agent acts on the answer.

Here's what the agent asks. First, after a search, it hands the chunks to the LLM and asks: do these actually answer the question?

If the answer is no, the agent asks the LLM for a better query ("claim days" might become "reimbursement claim deadline after discharge"), or for a different source to try, and runs the search again. Then the chunks get graded again.

Once they're good enough, the agent asks the LLM to draft an answer from them, and then to check that the draft is grounded: every claim in it has to point back to a chunk that says it. A claim that doesn't gets cut and the draft is written again. If the missing piece is something the question needs, the agent goes back to searching.

Follow the numbers in the figure below: two loops, with the agent in the middle of both.

None of these steps were invented last week. ReAct showed models alternating between reasoning and calling tools, looking at each result before deciding the next step. Self-RAG and Corrective RAG brought the same kind of self-checking to retrieval: grade what came back, search again when it's weak, and check the answer against the sources. What makes the system agentic is who decides when the evidence is good enough. In plain RAG, that's fixed in your code: one search, then answer. Here, the loop keeps going until the LLM judges the evidence good enough, or the agent runs out of budget.

Back to that hospital bill

Here's our claim question going through the loop.

1question  How many days do I have to submit a health insurance claim after discharge?
2
3search 1  "health insurance claim days after discharge"
4  found   network hospitals, cashless vs reimbursement, room rent limits, ...
5  grade   none of these mention a deadline -> not enough
6
7search 2  "reimbursement claim deadline after hospital discharge"
8  filter  doc_type = policy, year >= 2024
9  found   Claims SOP 4.2: "submit within 15 days of discharge ..."
10          Claims SOP 4.3: "attach the discharge summary and itemised bill ..."
11  grade   answers the question -> enough
12
13draft     "You have 15 days from discharge to submit a reimbursement claim.
14           Attach the discharge summary and the itemised bill."
15  check   "15 days"            -> Claims SOP 4.2   supported
16          "discharge summary"  -> Claims SOP 4.3   supported
17          "itemised bill"      -> Claims SOP 4.3   supported
18
19answer    15 days, with links to 4.2 and 4.3

The first search was a dud. Plain RAG would have answered from those hospital chunks and probably filled in the deadline itself, the 30 days from earlier. The agent noticed, searched with better words, and only answered once every sentence had a source behind it.

Why not let it loop forever?

If searching again gives better answers, why not let it keep going until it's sure?

Because every round costs something. Each grade, rewrite and draft is another call to the model, which means more tokens (that bill again) and more seconds of your user staring at a spinner. And some questions simply don't have an answer in your documents. An agent looking for something that isn't there will keep rephrasing the same question for as long as you let it.

So you give the agent a budget: three searches, say, or a fixed number of model calls, or a time limit. When the budget runs out, it stops, answers with what it has, and says plainly what it couldn't find:

1I couldn't find a submission deadline in the current claims policy. The closest match
2is the 2021 policy, which says 30 days, but that may be out of date. HR can confirm.

Remember where we started. The model's problem was that it wouldn't say "I don't know"; it filled the gap with something that sounded right. All of this, the chunks, the vectors, the grading, the grounding checks and the budget, is what finally lets it say "I looked here and here, and it isn't there." For someone trying to get a hospital bill paid, that's worth more than a confident guess.

The part I left out

Everything our agent did so far was read-only. It searched, it read, it answered. Now imagine the next thing someone asks for: "Can it just file the claim for me?"

Once an agent can change things, a wrong decision turns into a wrong action: a claim filed for the wrong amount, or a record updated that shouldn't have been. That's where you want a person in the loop, with the agent doing the legwork and a human approving anything that can't be undone. How to design that, permissions, approvals and all, is the next article. Subscribe to the newsletter so it lands in your inbox.