Lessons · LLM01: Prompt Injection · error
Whoever wrote a document your app retrieves can give orders to your model.
Caught by untrusted-retrieval-context-in-system-role · CWE-77, CWE-94
Your app looks things up (in a vector store, a search index, a web page, an inbox) and hands the results to the model so it can answer with them.
Here, those results go into the system prompt. That is the slot the model treats as your rules, so the retrieved text gets the same authority you have.
It feels safe because the text came from your own index. But you did not write most of what you indexed. A support ticket, a scraped page, or an uploaded PDF can carry instructions, and the model will read them as orders.
Your docs assistant searches uploaded files and puts the best matches into the system prompt as context.
const docs = await search(question);
await generateText({
model,
system: `Answer using these documents:\n${docs.join("\n")}`,
prompt: question,
});
The cost: The attacker never talks to your app directly. Anyone who can get text into your index can steer every user who triggers it.
Paste this into Claude Code, Cursor, or whatever wrote the code. When llm-audit finds the mistake in your project, npx llm-audit prompt gives you the same prompt with every file and line filled in.
Fix this in my project: retrieved documents or search results are placed in the model's system prompt, where they count as instructions.
Please:
1. Keep the system prompt a fixed string that contains only my instructions.
2. Put the retrieved text in a "user" message, wrapped in clear delimiters such as <documents> and </documents>.
3. Add one line to the system prompt telling the model that anything inside those delimiters is data to quote from, never instructions to follow.
4. If the model has tools, do not let text from retrieved documents trigger a tool call without a check.
Show me the diff.A prompt that looks for the same mistake everywhere else, without changing anything yet.
Check this project for retrieved text that reaches my AI model as instructions, before I ship it.
Look for:
- Search results, vector store matches, fetched web pages, emails, or uploaded files placed in the system prompt or system role.
- Retrieved text pasted into a prompt with no delimiters marking it as data.
- Places where retrieved text can lead to a tool call.
For each one, give me the file and line, say who could write the text that gets retrieved, and explain the fix. Do not change any code yet.npx llm-audit scan
Runs on your machine with Semgrep. Nothing is uploaded.