Securing AI applications

Why LLM apps need their own security thinking

Ordinary software separates code from data. A database query and the text a user typed travel in different channels, so with the right care the text can never become a command. A language model has no such wall: your rules, the user's question, retrieved notes and tool results all arrive as one stream of tokens, and the model may follow an instruction wherever it appears.

That single fact explains most AI-specific attacks. There is no known way to make a model reliably ignore instructions hidden inside the data it reads, so defences aim to reduce the chance of an attack working and to limit the damage when one does.

Classic security still applies on top. Keep API keys on the server and out of the browser, require sign-in for anything personal, rate-limit requests so nobody can run up your bill, and keep your packages up to date. An AI feature does not excuse a weak login page.

Prompt injection, direct and indirect

Direct injection is the user attacking your prompt. They type something like an order to ignore previous instructions, pretend to be the developer, or wrap a harmful request in a role-play. The goal is usually to extract the system prompt, bypass a rule or make the app produce content you would not want under your name.

Indirect injection is more dangerous because the attacker never talks to your app. They plant instructions in content the model will read later: a shared note, a web page your agent visits, an email it summarises, a document uploaded by someone else, or the output of a tool. When Study Buddy searches notes that a classmate shared, one of them could contain a hidden line telling the tutor to send the learner to a phishing site.

Several layers help, and none is enough alone. Mark untrusted content clearly, for example by wrapping it in tags and telling the model it is data. Keep untrusted text out of the system message. Check outputs before acting on them, and never let a single injected sentence trigger an action with real consequences.

Some teams add a separate classifier or a second model call that screens input and output for injection attempts. This catches common patterns but not clever new ones, so treat it as a filter that lowers noise, not as a guarantee.

Keeping secrets and personal data in

Assume the system prompt will leak. Determined users can usually coax it out, so never place keys, passwords, internal URLs or anything else confidential in it. If the prompt is valuable, its value should come from your product around it, not from secrecy.

Anything in the context window can appear in the output. If retrieval pulls one learner's notes into another learner's request, the model may quote them. Filter by user and permission before the search runs, not after the answer is written, so data the person may not see never reaches the model at all.

Be careful how you render model output. If your interface turns model text into clickable links or loads images from addresses the model wrote, an injected instruction can make the model build an address that carries private data to an attacker's server. Show links as plain text or allow only trusted domains.

Finally, think about what leaves your system. Send providers only the data a task needs, check their data retention terms, remove names and identifiers where you can, and keep your own logs as protected as the database they describe.

Least privilege for tools

Tools turn a text problem into a real-world one. A fooled model that can only read the study plan is an annoyance; a fooled model that can delete files or send emails is an incident. Design every tool on the assumption that the model calling it may have been manipulated.

Give each tool the smallest power that does the job, and the narrowest credentials behind it. Prefer read-only tools, scope them to the current user, validate every argument in code, and keep an allowlist of the actions the model may request.

• Read-only by default; add write access only where the feature needs it
• Credentials scoped to one user and one purpose, never an admin key
• Human confirmation before sending, deleting, paying or publishing
• Limits on how many calls and how much spend one session can trigger
• Logs of every tool call with its arguments and result
• Extra caution with third-party tools and MCP servers you did not write

Red-teaming your own app

Red-teaming means attacking your system on purpose, the way a real adversary would, before release. For an LLM app that covers security failures such as leaks and tool misuse, and also harmful or embarrassing content that ordinary users might trigger by accident.

Start small and keep it repeatable. Write a list of attack prompts, plant a fake secret called a canary in the system prompt, and check automatically whether it ever shows up in a reply. Add injected notes to your test data and check that the tutor treats them as content. Run the suite after every change to the prompt, the model or the tools, because a fix in one place often opens a gap in another.

Public catalogues help you think wider. The OWASP Top 10 for LLM applications lists the most common weaknesses, and MITRE ATLAS collects techniques used against AI systems. Both cover risks beyond injection: poisoned training or retrieval data, compromised models and packages, and people relying on wrong output without checking it.