Prompt Injection

When user-supplied text (or text fetched from a website, document, or email the model reads) is placed into the same context window as the system's actual instructions, an attacker can write text that the model treats as a new, overriding instruction — hijacking the assistant's behavior.

Direct prompt injection is the user typing an override straight into the chat; indirect prompt injection is the payload hidden in a document, webpage, or email the model is asked to summarize or process

There is no known way to make an LLM perfectly immune to this at the model level — defense has to happen in the surrounding application (what the model is allowed to do, not just what it's told to do)

Treat anything the model reads that wasn't written by a trusted operator — user messages, retrieved documents, tool outputs — as untrusted input, exactly like an HTML form field

The blast radius of a successful injection is bounded by what tools/actions the model has access to — the real fix often lives in limiting tool permissions, not just in the prompt

Want this flow explained in detail as it plays?

POST/api/chat

Headers

Host:app.flowground.dev

Content-Type:application/json

Body

Quick-Insert Payloads

Defense in depth

Structurally separate trusted system instructions from untrusted content wherever the model API supports it, and never concatenate user/document text directly into the system prompt string
Apply the principle of least privilege to any tool the model can call — a prompt injection that convinces the model to 'send an email' is harmless if the model has no send_email tool available at all
Add a human-in-the-loop confirmation step for any action with real-world side effects (sending money, sending email, deleting data) that a model proposes, regardless of how it was convinced to propose it

Vulnerable vs. Fixed

const systemPrompt = "You are a helpful support assistant.";
const userTicket = req.body.ticketText; // untrusted — may contain hidden instructions
const response = await llm.chat({
  messages: [
    { role: "system", content: systemPrompt + "\n\nTicket: " + userTicket }, // concatenated into the trusted instruction channel
  ],
  tools: [sendEmailTool, refundTool], // full tool access, no gating
});