Prompt Injection in Retrieval-Augmented Generation Systems

Retrieval-augmented generation was supposed to make large language models more trustworthy. Pulling facts from a real document store instead of guessing from memory shrinks hallucinations. What it actually did was open a second front door. The retrieval pipeline itself, the documents, the ranking step, the context window, is now a place where an attacker can plant instructions the model will treat as fact, and sometimes as commands. Patching the model doesn't close that door, because the model was never the component that broke.

A RAG model has no way to tell a system instruction from a sentence in a document someone else wrote. Every token in the context window gets treated the same: the system does not distinguish tokens from the developer's prompt from tokens in a wiki page an attacker edited last Tuesday. A recent taxonomy paper (arXiv:2604.08304) frames the real boundary this way: it runs between what the model learned during training and whatever the retrieval pipeline just handed it moments ago. That's the fight now, and an attacker never has to touch the model, the API, or the user's prompt to win it. Poison the corpus, tilt the ranking, or manipulate what gets disclosed, and the job's done before anyone types a question.

Researchers at Hong Kong Polytechnic and HKUST broke the RAG pipeline into six operational stages, then grouped the threats against those stages into four surfaces worth knowing by name (arXiv:2604.08304):

Pre-retrieval corruption poisons the knowledge base before a single query is ever issued. Retrieval-time manipulation games the ranking and selection logic at query time, pushing bad documents into the results an attacker wants surfaced. Downstream context exploitation triggers instructions hidden in retrieved chunks once they land inside the generation step. Knowledge exfiltration turns the retrieval channel itself into the exit route for data leaving the system.

Most defenses on the market handle exactly one of these and call it a day. Harden the model against jailbreaks, and the corpus sits there unpoliced. Adding a filter on retrieved documents does nothing to stop generation-time injection, which sails right past it. No widely deployed defense watches all four at once, and that gap is where the real attacks live.

What makes this worse than a normal prompt injection against a plain chatbot is that the damage doesn't stop at one conversation. That kind of attack is a one-off: someone crafts a bad prompt, gets a bad response, and the damage stops at that conversation. Poison a RAG knowledge base, though, and the payload just sits there. It survives across sessions, hits every user whose query happens to retrieve it, and never announces itself as an incident anyone would think to investigate. That's a different category of bug: persistent, shared across a whole user base, and nearly impossible to trace back to a single cause.

Knowledge base poisoning: how an attacker controls the corpus before any query arrives

The attack model here is almost boringly simple. Get a document into the repository the RAG system pulls from, wait, and let a user's query do the rest. No live interaction needed. The payload sits dormant until someone asks a question that happens to retrieve it.

PoisonedRAG gave this a name and a number. In a black-box setting, researchers injected five malicious documents into a knowledge base holding 2.6 million texts and hit a 97% attack success rate on one question-answering benchmark, 99% on another, and 91% on a third. Five documents, out of 2.6 million. That ratio should change how anyone thinks about corpus size as a defense, because size alone buys nothing. The USENIX paper behind the result also found that the defenses tested against it weren't good enough to matter.

CorruptRAG pushed the constraint further: one document instead of five. And one variant of the attack does something almost administrative in its cunning. It builds a document formatted like a routine correction notice, labeling the "old" answer as outdated and presenting the attacker's target answer as the current, corrected fact. It reads exactly like the kind of update a real knowledge base would contain on any given Tuesday. No broken syntax, no red flags. Just a fake correction sitting where a real one might sit.

UniC-RAG, out of Penn State (arXiv:2508.18652), takes the idea further still by generalizing across topics instead of targeting one question at a time. Researchers optimized a batch of just 100 adversarial texts, dropped them into a database holding millions of entries, and used them to attack 2,000 different queries across a wide range of subjects at once. Success rate: over 90%. The payoff isn't limited to wrong answers, either. The same mechanism can redirect users to malicious sites, trigger commands the system was never meant to run, or just grind retrieval to a halt as a denial-of-service play.

Corpus size was never the defense people assumed it was. A hundred bad documents buried in millions of good ones still win, and win consistently.

Retrieval-time manipulation: exploiting the ranking step before the model ever reads a document

Poisoning the corpus is one path in. Gaming the ranking step is another path in, and it doesn't require touching the document store.

Think of it as a two-part exploit. First, the neural ranking model, the thing deciding which of a million candidate documents makes it into the top results, gets manipulated into surfacing something it shouldn't. Second, that document, now sitting inside the context window, delivers its payload during generation. Vector, then payload. It's a familiar shape from other corners of security, just transplanted into retrieval.

What sets this apart from corpus poisoning: the document doesn't need to contain malicious content. Craft a query, or craft a document's embedding, so it scores unusually well against the retriever's similarity function, and a mediocre or misleading document can outrank the accurate one on vector math alone.

The SafeRAG benchmark sorted these attacks into distinct categories, including noise injection, inter-context conflict, and denial-of-service variants. Tested across a range of representative RAG components, the finding was uncomfortable. Even the crudest, most obvious versions of these attacks slipped past existing retrievers, filters, and advanced language models alike. SafeRAG's findings speak to how widespread this exposure already is. Any enterprise running RAG in a regulated industry, in any language, inherits the same structural gap. Nothing about it is language-specific or region-specific once you look at the mechanism.

Then there's a paper with a blunt title, "Machine Against the RAG," describing a blocker-document technique. Instead of corrupting the answer a system gives, it introduces documents designed to degrade retrieval quality and deny useful service. Different goal, same surface. Most defenses built to catch misinformation injection aren't even looking for this kind of attack.

Indirect prompt injection at generation time: when retrieved content becomes executable instruction

Indirect prompt injection, IPI for short, is the mechanism tying all of this together at the point where the damage actually lands. A hidden or manipulated instruction sits inside content, a web page, a document, an email, and waits to be retrieved. The user asking the question did nothing wrong. They never saw the payload. It arrived through the retrieval path, not through anything they typed.

Severity isn't fixed here. It depends on what the model is allowed to do once it reads the poisoned content. A read-only assistant that just answers questions has a narrow blast radius. An agentic system wired up with tool access, one that can send emails, hit APIs, or move files, has a much wider one. The same injected instruction means something completely different depending on which kind of system swallows it.

One finding stands out. Attacks that phrase their instructions in natural, domain-appropriate language, rather than blunt override commands like "ignore previous instructions," slip past detection at rates above 90%. In one evaluation, production-grade classifiers, including Llama Guard 3, caught every static override-directive attack thrown at them, and detected zero of the camouflaged, domain-fluent versions. Researchers called this gap the Camouflage Detection Gap, and it ran between 80 and 100% across the models tested. A detector failing completely, not partially, against an attack style it was never built to recognize: that's the entire point of the gap.

Most documented real-world cases before this were low-stakes and almost mischievous. Hidden "hire me" text buried in resumes to trick AI screening tools, anti-scraping messages planted in web pages, the occasional bit of self-promotion slipped into content meant for an AI reader. Then Unit 42 documented a malicious IPI attack built specifically to slip past an AI-based ad review system, demonstrating that the technique works against real production filters. That was the shift from theoretical risk to demonstrated abuse.

EchoLeak: what the first weaponized production case reveals about structural exposure

Aim Security disclosed EchoLeak in June 2025, and it's the case that turned this from an academic worry into a documented production exploit. The target was Microsoft 365 Copilot. The attack required zero clicks from the victim.

The chain worked like this. An attacker sends an email that looks entirely ordinary. Buried inside it, invisible to a human reader, sits a hidden prompt: white text on a white background, or text tucked into an HTML comment. The user never sees any of it. But Copilot's language model does, because at some later point, when that same user asks Copilot an unrelated question, the RAG engine retrieves the old email while building context for the answer. The hidden payload rides along, and it executes during generation.

Getting there took stacking several bypasses, not just one. The attack evaded Microsoft's XPIA classifier, built specifically to catch cross-prompt injection. It got around link redaction by using reference-style Markdown formatting instead of a direct link. It exploited how Copilot auto-fetches images. And it abused a Microsoft Teams proxy that the content security policy happened to allow through. Chain all four together, and the result was full privilege escalation across the model's trust boundaries. Copilot reached into internal files it had access to and shipped their contents to a server the attacker controlled. Zero user interaction, start to finish.

The affected surfaces ran across Word, Excel, PowerPoint, Outlook, and Teams integrations, basically the whole Microsoft 365 Copilot footprint. And here's the detail that should sit uncomfortably with anyone running traditional security tooling: the payload was plain text. No malware signature, no executable, nothing an antivirus scanner or a firewall would ever flag. Copilot wasn't malfunctioning. It was doing what it was built to do, reading retrieved content and acting on it, and that's the design assumption the whole attack exploited.

Microsoft patched the vulnerability server-side and said there was no evidence of exploitation in the wild. But EchoLeak matters even if no one exploited it maliciously before the patch landed. It's the first documented case of prompt injection weaponized for actual data exfiltration inside a production AI system, and it confirmed, in a live enterprise product, everything the academic poisoning and injection research had been warning about for months.

Limits of existing RAG defenses

Lining up the four surfaces against the defenses currently on the market reveals a pattern fast: every defense covers one surface and leaves the other three wide open. Most teams still get that backwards. They ship one fix, call the system hardened, and move on.

Fine-tuning the model for robustness, or bolting on an instruction hierarchy, helps against some prompt-level attacks. Sure. But it does nothing for a poisoned corpus, and nothing for a retriever that's been gamed into surfacing the wrong documents. It comes with a real cost too: added latency, worse performance on legitimate queries, and a false sense that the problem's handled just because the model layer looks tougher.

Redesigning the retriever to screen out suspicious documents at query time is a genuine improvement, but only when the bad document actually looks suspicious. A single-document poisoning attack engineered to blend in with legitimate content, the constraint CorruptRAG demonstrated, looks unremarkable. It looks exactly like legitimate content, because that's the whole design of the attack. Should a defense built to spot outliers even be expected to catch something engineered to look ordinary? It shouldn't, and that's the gap.

Classifier-based filtering runs into the same wall. The domain-camouflage research (arXiv:2606.18530) puts a number on it: a Camouflage Detection Gap of 80 to 100%. Classifiers trained to catch phrases like "ignore previous instructions" are looking for a syntax pattern. Payloads written in fluent, domain-native language don't carry that pattern, so the classifier has nothing to catch. Llama Guard 3, tested directly, successfully detected static override-directive attacks while missing every camouflaged one.

None of this means defense is hopeless. It means defense built around one stage of the pipeline solves one stage of the problem, no more. Corpus integrity, retrieval-time scrutiny, and generation-time monitoring of retrieved-content behavior are three separate jobs. A RAG system that only does one of them is, structurally, only a third defended. Anyone securing a RAG deployment has to reckon with the whole lifecycle at once, including the parts that resist a quick patch, because that's exactly where EchoLeak and PoisonedRAG both found their opening.