How AI Agents Retrieve External Knowledge at Runtime

Large language models freeze their knowledge the moment training ends. Ask one about something that happened after that cutoff, and it either admits it doesn't know, or worse, guesses with total confidence. Retrieval-augmented generation exists to hand a model facts it never learned, right when it needs them.

Three failure modes fall out of that frozen state, and they build on each other rather than staying separate. Stale knowledge is the obvious one, since a policy changes, a product ships a new version, an event happens, and the model just doesn't know, because none of it existed when the training data was collected. Hallucination is the more dangerous cousin. When a model lacks ground truth, it doesn't stop and say so. It generates something plausible-sounding and made up, because that's what a next-token predictor does when it hits a gap. Then there's the blind spot around private data: internal wikis, proprietary databases, niche technical corpora. None of that was ever public, so none of it was ever in training, no matter how recent the model is.

Retraining looks like the obvious fix until someone actually runs the math on it. Full retraining produces just another frozen snapshot with a later cutoff date. The same problem resets on a delay. Fine-tuning gets mistaken for a fix too, but it can't teach a model to cite a document it never saw. So any agent working in a live environment, one with shifting products, active regulations, real-time events, needs a way to pull knowledge in at the moment it answers a question, not bake it into weights ahead of time. Static documentation only makes this worse: knowledge sitting in a PDF, a wiki, or a siloed internal database might as well not exist unless something actively connects it to the model at query time.

How classic RAG addresses the frozen-knowledge problem

Retrieval-augmented generation, RAG for short, handles this with a mechanism that's almost mechanical in its simplicity: find text relevant to the question, stick it into the prompt, let the model answer using what it was just shown instead of what it memorized months ago.

The pipeline runs in a fairly fixed order. Documents get broken into smaller chunks, since a whole manual or PDF is too big to search efficiently or fit into context. Each chunk gets turned into a vector embedding, a numerical stand-in for its meaning. When someone asks a question, that query gets encoded the same way, and the system compares it against every stored chunk by similarity. The top-k matches, whatever k is set to, get pulled and dropped into the model's context window next to the original question. The model then answers based on both.

A few design choices decide how well this works in practice: how big each chunk is, which embedding model does the encoding, how many chunks come back per query. Get chunk size wrong and you either lose context (too small) or dilute relevance (too large). The real advantage over retraining is operational: the knowledge store updates on its own schedule. Add new documents, delete outdated ones, and the model itself never has to change.

Early retrieval leaned on sparse keyword methods like BM25, matching literal terms between a query and a document. Dense vector retrieval came in as an improvement, catching semantic similarity that keyword matching misses entirely, per the survey in arXiv:2407.13193. RAG's appeal to enterprises follows directly from all this: it cuts down on hallucination, it lets an answer point back to its source, and it works across different domains without anyone touching the model.

Where single-pass retrieval consistently breaks down

Classic RAG runs one query, does one retrieval pass, produces one answer. If that first retrieval misses the right chunk, nothing in the architecture catches it. There's no second try built in, and that gap is the actual weak point in the whole design, not some footnote worth a passing mention.

Research published in arXiv:2509.04820 found that 48% of traditional RAG failures traced directly to the model simply not finding the relevant chunk in its top-k results, in a benchmark built on government documents. Nearly half of all failures trace back to one structural weak spot. That gap amounts to more than a rounding error. That's the architecture telling you exactly where it breaks.

Multi-hop questions expose why. A question that needs evidence pulled from several disconnected passages needs multiple retrieval steps, because the query vector represents the question as asked, while the intermediate facts a person would need to string together require separate retrieval. Ask "how does policy X interact with regulation Y for case Z" and a single retrieval pass has no way to know it needs three separate documents, each answering a different piece.

Meta's CRAG benchmark puts a number on the ceiling: standard RAG lands around 44% accuracy, and even the better industry RAG implementations top out near 63%. For anything mission-critical, legal research, medical guidance, financial compliance, that gap is the difference between useful and unusable.

Recency is its own separate failure, and arguably the one that should worry people most. RAG ranks by cosine similarity, which rewards semantic closeness, not how new something is. Per arXiv:2605.17625, this produces solid results on stable historical facts but drops to 0% accuracy on "current state" questions, things like "what is the current threshold." The architecture isn't built to notice something changed last week. It isn't built to notice time at all.

And there's no memory layered on top. Every query starts cold, with no sense of what got retrieved or reasoned through in a prior turn. All these failures share one root cause: retrieval is a single mechanical step bolted onto generation, not part of how the model actually reasons. Fixing that means rewiring the loop, which is exactly where agentic approaches start.

One-shot retrieval improvements: wider nets before iterative loops

Before jumping to iterative loops, there's a simpler fix worth understanding first, since it's cheaper to build and it's usually where teams start. If a small net cast once misses the fish, the obvious move is to cast a bigger net, not necessarily a smarter one. That's the cheaper fix, and it's also the one teams lean on too long, mistaking it for a full solution when it's really just triage.

That's the logic behind One-SHOT adaptive retrieval, described in the same SenseTime paper (Lin et al., 2025). Instead of a fixed top-k, the system pulls in as many relevant chunks as fit inside a token budget, ranked by relevance per token so the densest, most useful passages get priority first. Rule-based filters cut down the noise that comes from casting that wider net.

Hybrid search runs alongside this as a separate improvement, and by now it's close to a default in production systems. It combines dense vector search, which catches meaning, with sparse keyword search like BM25, which catches exact terms, then fuses the two ranked lists together, typically through Reciprocal Rank Fusion. Practitioner benchmarks on enterprise document sets consistently show better recall from the combination than from either method alone. Onyx.app reported that enterprise intent to adopt hybrid retrieval tripled in a single quarter of 2025 as more RAG programs hit scale limits with vector-only search.

Corrective RAG and Self-RAG add a validation checkpoint on top of all this: retrieved chunks get scored for relevance before they pass to the generator, and low-confidence retrievals get thrown out instead of feeding into the answer.

But every one of these techniques shares the same ceiling, and it's worth being blunt about it: none of them touch the actual architecture. They improve what comes back from a single retrieval pass. None of them let the system reformulate the question, follow a chain of reasoning, or go looking for more once it sees what the first pass turned up. For that, a loop is required. Teams that stop here, convinced hybrid search solved their retrieval problem, are treating a symptom and calling it cured.

How agentic RAG embeds retrieval inside the reasoning loop

This is the real architectural break from everything above. Retrieval stops being a preprocessing step that happens once before generation starts. Instead, the model decides mid-generation that it needs more information, issues a query, looks at what comes back, and keeps reasoning from there, possibly firing off another query if the first answer still isn't enough.

The SenseTime paper calls this "multi-cast" fishing: smaller, targeted queries fired off repeatedly, each one shaped by what the previous retrieval turned up, gradually converging on the actual evidence needed instead of gambling everything on one big cast.

Making that loop work asks the model to take on several roles at once, laid out by Singh et al. (2025, arXiv:2501.09136). Query routing decides which knowledge source fits a given sub-question. Task decomposition breaks a hard question into smaller retrievable pieces. Tool use covers actually calling a search API, a database, or a structured store when needed. Stopping criteria decides when there's enough evidence gathered to quit looping and start answering.

A handful of methods make this interleaving possible in practice: ReAct (Yao et al., 2023), Self-Ask (Press et al.), and Search-o1 (Li et al., 2025), all of which let a model mix generation with retrieval by spotting its own knowledge gaps mid-answer and firing off a targeted query to fill them, per arXiv:2509.04820.

Two failure modes show up specifically once retrieval turns iterative, and the SenseTime paper designs directly against both. Query drift is when successive reformulations wander away from the original question, chasing tangents instead of the actual answer. The opposite problem is the model stopping its queries too soon and settling for partial evidence instead of pushing further.

This pattern is already running in production. OpenAI Deep Research, Gemini Deep Research, and Perplexity Deep Research all implement some version of it, per arXiv:2510.13910, models acting as agents that search, reason over what they find, and search again based on that reasoning. Agentic RAG systems with intelligent memory have been shown to outperform traditional RAG on accuracy, while also reducing token usage compared to full-context approaches.

The retrieval mechanisms agents actually call at runtime

Behind any of these agentic loops sits a set of concrete tools the agent actually calls, and they are not interchangeable. Knowing which one fits which job matters more than knowing that all of them exist.

Vector databases remain the workhorse for semantic retrieval, turning documents into high-dimensional embeddings and searching by similarity rather than exact keyword match. Pinecone is built for retrieval at very large scale. pgvector takes a different tradeoff, storing embeddings right next to relational data with full ACID guarantees, though it's not the tool to reach for once vector counts get into the truly massive range. The production default now is hybrid, dense and sparse search fused together with Reciprocal Rank Fusion, rather than leaning on dense vectors alone.

Knowledge graphs solve a different kind of problem: relational reasoning that vector similarity just isn't built for. Microsoft Research's GraphRAG builds a multi-layered knowledge graph out of a document set, identifies hierarchies of entity communities, and uses that structure to generate grounded summaries. On schema-bound queries involving relationships, KPIs, or hierarchical context, the gap is stark: FalkorDB's 2025 benchmarking (via the Salfati Group) found vector-only RAG scoring 0% on some of these queries while optimized GraphRAG scored well above 90%. The tradeoff is cost: indexing a graph runs orders of magnitude more expensive than standard vector RAG, though Microsoft Research's LazyGraphRAG cuts that cost substantially. The GraphRAG project itself is largely in maintenance mode as of its GitHub repository, not taking new pull requests or new features going forward. The practical pattern that emerges is routing: simple factual queries go to vector RAG, complex relational ones go to GraphRAG, decided automatically through query classification.

Live web search fills a gap neither vector databases nor knowledge graphs can: information that didn't exist when any index was built. Exa and Tavily are AI-native search APIs, built to understand semantic intent and return results already formatted for a model to consume rather than for a human to click through. Exa, which raised a large Series A round in 2025, pushes this further: its Search API takes a plain natural-language objective and returns excerpts optimized for an LLM, replacing what used to be several separate keyword searches with one intent-driven call. Microsoft shut down its Bing Search APIs on August 11, 2025, pushing users toward Azure AI Agents and its "Grounding with Bing Search" feature instead. Research consistently shows standard LLMs doing basic keyword search performing poorly on complex multi-hop research benchmarks, while iterative retrieval systems score substantially higher. Teams keep shopping for the "best" search API when the actual gap sits one level up, in the loop wrapped around it. The API matters less than what calls it, and when, and how many times.

Agent memory stores round out the picture, handling retrieval from what the agent itself already gathered or reasoned through earlier. The hierarchy roughly mirrors human memory: short-term for the current conversation, working memory for active reasoning, long-term memory split into episodic, semantic, and procedural. Semantic memory usually runs on the same knowledge bases and vector databases already discussed, often merged directly with RAG to add domain expertise without retraining anything. MemGPT (Packer et al., 2023) proposed a memory hierarchy specifically to get around limited context windows, pulling in from long-term storage and dropping what isn't currently relevant. Frameworks like Mem0, Letta, Zep, and LangMem have shown up to handle the extraction, consolidation, and retrieval work this requires.

The limitation worth carrying forward: RAG's 0% floor on current-state queries exists because memory stores, like static indexes, don't automatically know what changed. That's a documentation problem as much as a retrieval one, and it's where the next section picks up.

MCP as the standardization layer that makes tool-calling coherent

None of the retrieval mechanisms above matter to an agent unless something lets it call them consistently, which is what the Model Context Protocol was built to solve.

Before a shared protocol existed, connecting M agent applications to N different data sources meant building M×N custom integrations, a mess that got worse with every new tool or every new agent added to the mix. Anthropic introduced MCP in late 2024 as a standardized interface, defined through JSON-RPC message exchange, so that connecting the same M applications and N sources turns into M+N integrations instead. At scale, that's the difference between a linear problem and a quadratic one, and it's not a small difference.

Adoption moved fast. Within a few months of release, over a thousand community-built MCP servers were already available, and by 2025 the protocol had native support across major ecosystems, OpenAI, Anthropic, and Google among them. In practical terms, one agent can call a vector database, a knowledge graph, a live search API, and a memory store all through the same interface, routing retrieval decisions on the fly without a custom connector built for each one.

MCP doesn't touch the freshness problem on its own, though, and that limitation needs to be stated so nobody mistakes standardization for a cure. It standardizes how an agent reaches a knowledge source, but says nothing about whether that source is actually current, accurate, or well organized. Standardized access and good content are two separate problems, and solving the first doesn't quietly solve the second. That distinction carries the whole argument of the final section.

Why the quality of retrieved knowledge depends on how documentation is maintained

The temporal failure documented in arXiv:2605.17625, RAG scoring 0% on current-state questions, points to more than an architecture problem. It's a documentation problem wearing an architecture costume. Retrieval techniques, however clever, can only surface information that got updated in the index to begin with, and no amount of hybrid search or GraphRAG routing changes that.

RAGFlow's 2025 year-end review captured this with a phrase worth sitting with: enterprises describe RAG as something they "cannot live without, yet remain unsatisfied" with. Dig into why, and the gap usually traces back to something other than the retrieval mechanism itself. It's the underlying knowledge quality and freshness feeding that mechanism, a piece most teams never budget for.

Static, siloed documentation is where this actually breaks down day to day. A PDF that never gets re-ingested after a product update. A wiki that quietly drifts out of sync with what actually shipped. Internal knowledge written for humans to skim, structured in a way a retrieval system simply cannot search well. All of this produces failures that look, on the surface, like the model got something wrong. Really, the model answered correctly based on what it was given, and what it was given was stale. Blame the pipe, not the water running through it.

RAGFlow's 2025 review points to a broader shift underway: away from optimizing individual retrieval algorithms in isolation, toward designing the entire pipeline end to end, retrieval, context assembly, model reasoning, as one connected system. Documentation structure sits inside that pipeline. It isn't a separate concern bolted on afterward, and treating it as one is exactly where most teams go wrong.

What that means concretely: documentation needs to update in step with product changes, not on whatever manual publishing schedule a team happens to keep. It needs structure that makes content genuinely searchable, not paragraphs of prose with the useful facts buried three sentences deep. And it needs to plug into the agent's retrieval layer automatically, so the system points at what's current rather than what was true six months ago.

Teams pouring money into retrieval architecture, agentic loops, hybrid search, GraphRAG routing, MCP integration, while leaving the underlying documentation static are solving the easier half of the problem. The frozen-knowledge issue that made RAG necessary in the first place just resurfaces one layer up, at the documentation layer instead of the model layer. Accuracy in an agentic system depends on more than how sophisticated the retrieval is. It depends just as much on whether what's being retrieved is actually still true, and that second half is the one worth spending money on first.