Prompt Injection: Protect Your RAG and Agents

When a language model reads text, it cannot reliably tell the difference between instructions from you and instructions hidden in the content it is processing. That weakness is called prompt injection, and it is the first entry in the OWASP Top 10 for large language model applications. It matters most when an AI system reads outside content, such as emails, web pages or uploaded files, and can take actions, such as sending messages or updating records. There is no complete fix today, but there are well-understood ways to limit the damage. This guide explains how prompt injection works, why retrieval-augmented systems are exposed and which design choices reduce the risk.

What prompt injection is

The OWASP page on the risk describes a prompt injection vulnerability as occurring when user prompts alter the model's behaviour or output in unintended ways, and notes that the inputs can affect the model even if they are imperceptible to humans, as long as the model parses them (OWASP: LLM01 Prompt Injection). It is listed as LLM01 in the OWASP Top 10 for LLM applications (OWASP Top 10 for LLM applications).

OWASP distinguishes two types:

• Direct prompt injection, where a user's own input changes the model's behaviour, intentionally or by accident.
• Indirect prompt injection, where the model accepts input from external sources, such as websites or files, and that content contains instructions that change its behaviour.

Why RAG systems are exposed

Retrieval-augmented generation, or RAG, works by fetching documents and placing them in the model's context so it can answer from them. That is exactly the route indirect prompt injection uses. A document in your knowledge base, a web page fetched by an agent or an email being summarised can contain text written to look like instructions.

OWASP notes that while techniques like RAG and fine-tuning aim to make outputs more relevant and accurate, research shows they do not fully mitigate prompt injection. In other words, adding your own documents does not make a system safer against this risk; it can add new sources of untrusted text. Our guide to private RAG covers how to keep document search within the right permissions, which limits who can plant content in the first place.

What attackers try to achieve

The consequences depend on what the system can do. Common goals include:

• Getting the model to reveal its instructions or data it should not share.
• Changing answers, such as inserting false information or links.
• Triggering actions, such as sending an email, changing a record or calling an API.
• Bypassing content rules set by the application.

The more access and autonomy a system has, the more an injection can do. A chatbot that only answers questions is far less exposed than an agent that can send email on your behalf.

Limit what the model can do

The most effective defences limit impact rather than trying to detect every malicious prompt. OWASP's mitigation list includes enforcing privilege control and least-privilege access: give the application its own API tokens for extended functions, handle those functions in code rather than giving them to the model and restrict the model's access to the minimum it needs.

In practice:

• Give tools narrow scopes, such as read-only access to one module rather than full CRM access.
• Keep sensitive operations in ordinary application code with its own checks.
• Separate systems that read untrusted content from systems that can take powerful actions.

Our guide to AI agent guardrails covers how to define these limits for agents.

Keep a person in the loop for risky actions

OWASP also recommends requiring human approval for high-risk actions, using human-in-the-loop controls for privileged operations. Sending external emails, issuing refunds, deleting records and changing permissions are good candidates. A clear approval step, showing exactly what will happen and why, stops an injected instruction from turning directly into an action.

Separate and label untrusted content

Another OWASP recommendation is to segregate and identify external content, clearly marking untrusted content to limit its influence. When building prompts, put retrieved documents in a clearly delimited section, tell the model that the section contains data rather than instructions and never place untrusted text where system instructions belong. This does not make injection impossible, but it reduces how often it works.

Validate outputs with code

OWASP suggests defining expected output formats and using deterministic code to check that responses follow them, along with input and output filtering for sensitive content. If a model is supposed to return a category from a fixed list, reject anything else. If it proposes an action, check the action against rules in code before executing it. Treat model output as untrusted input to the rest of your system.

Worked example: a risk review for one assistant

The table below is a hypothetical review of an internal assistant that summarises customer emails and can draft replies and update CRM fields.

CapabilityUntrusted input involvedRisk if injectedControlSummarise incoming emailEmail body and attachmentsMisleading summaryShow source email alongside summaryDraft replyEmail contentReply with false claims or linksHuman reviews and sends; links checked against allow listUpdate CRM status fieldEmail contentWrong status setAllowed values validated in code; change loggedLook up order historyCustomer ID from emailData from another customerLookup limited to the sender's verified accountSend email directlyNot permittedNot applicableCapability not given to the model

The last row is often the most important decision: some capabilities are safer not offered at all.

Test with adversarial inputs

OWASP's list also includes adversarial testing and attack simulations. Build a set of test documents and messages containing injection attempts, such as hidden instructions, requests to ignore previous rules and instructions disguised as formatting, and run them against your system regularly, especially after changing prompts, models or tools. Record what got through and adjust controls.

Monitor and respond

Log prompts, retrieved sources, model outputs and actions taken, with appropriate privacy controls. When something unexpected happens, these logs show which content was involved. Set alerts on unusual patterns, such as a sudden rise in tool calls or attempts to reach data outside the expected scope.

Frequently Asked Questions

What is prompt injection?

It is when text supplied to a language model, directly by a user or indirectly through documents, web pages or emails, changes the model's behaviour in ways the application did not intend.

Can prompt injection be fully prevented?

OWASP says it is unclear whether fool-proof prevention exists. The practical approach is to limit what the model can do, require approval for risky actions and validate outputs.

Does RAG make prompt injection worse?

RAG places retrieved content in the model's context, which can carry indirect prompt injection. OWASP notes that RAG and fine-tuning do not fully mitigate the risk.

What is the most important defence?

Limiting privileges. A model that cannot take harmful actions, or needs human approval to do so, limits the damage an injection can cause.

Build AI systems with sensible limits

We design RAG and agent systems with least-privilege tools, approval steps and output validation built in. See our AI agent orchestration and RAG service, or talk to us about prompt injection risks.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.