Lessons · LLM01: Prompt Injection · error
Whoever types into your app can rewrite the rules your model follows.
Caught by untrusted-input-in-system-prompt untrusted-input-concatenated-into-prompt-template · CWE-77, CWE-94
A model call usually has two kinds of text. The system prompt is yours: it sets the rules. The user message is theirs: it is the question. The model gives the system prompt more weight.
Here, text that came from outside your app (a form field, a URL parameter, a request body) is pasted into the system prompt or into one long prompt string. The model can no longer tell your rules apart from their words.
There is no escaping function that makes this safe. Language models do not have a reliable way to ignore instructions that appear in the instruction slot.
Your support bot builds its instructions from the customer's company name, so replies feel personal.
messages: [
{ role: "system", content: `You help customers of ${req.body.company}. Never give refunds.` },
{ role: "user", content: req.body.message },
]
The cost: The attacker controls what your model does. How bad that is depends on what the model can reach: data, tools, or other users.
Paste this into Claude Code, Cursor, or whatever wrote the code. When llm-audit finds the mistake in your project, npx llm-audit prompt gives you the same prompt with every file and line filled in.
Fix this in my project: untrusted text is put into the model's system prompt, or into a single prompt string that mixes my instructions with the user's text.
Please:
1. Keep the system prompt a fixed string that contains only my instructions.
2. Move the user's text into a separate message with role "user".
3. Validate the incoming request body with a schema (zod is fine), with a maximum length on free-text fields.
Show me the diff and explain in two sentences what an attacker could no longer do.A prompt that looks for the same mistake everywhere else, without changing anything yet.
Check this project for places where text from users can change my AI model's instructions, before I ship it.
Look for:
- System prompts or instructions built with template strings that include request data, form fields, URL parameters, or database values users can edit.
- One combined prompt string that mixes my instructions with the user's text, instead of separate system and user messages.
For each one, give me the file and line, show what an attacker would type to take over the instructions, and explain the fix. Do not change any code yet.npx llm-audit scan
Runs on your machine with Semgrep. Nothing is uploaded.