Sensitive Information Disclosure via LLM

LLMs are cooperative by default — if sensitive information is present anywhere in the context window (a system prompt with internal logic, another user's data pulled in by a buggy retrieval step, an API key embedded in a tool's instructions), a sufficiently persistent user can often extract it through creative rephrasing, even when the model was told not to share it.

A system prompt is a suggestion, not a security boundary — anything you don't want a user to ever see should not be placed in the model's context at all

Never embed secrets (API keys, internal URLs, other users' data) in a prompt and rely on the model 'choosing' not to repeat them — assume anything in context can eventually be extracted

Retrieval-augmented generation (RAG) systems need the SAME per-user authorization checks on retrieved documents that any other data-access code path would need — the model doesn't know it's not supposed to read tenant B's file

Output filtering (scanning the model's response for known secret patterns before it's returned) is a reasonable last line of defense, not a substitute for keeping secrets out of context in the first place

Want this flow explained in detail as it plays?

POST/api/chat

Headers

Host:app.flowground.dev

Content-Type:application/json

Body

Quick-Insert Payloads

Defense in depth

Never place secrets, internal-only business logic, or another user's data into the model's context window — keep those entirely out of what the model can see, rather than instructing it not to share them
Enforce the same row-level/tenant-level authorization on documents fetched for RAG that you would enforce on any other API returning that data — filter BEFORE the documents reach the model, not after
Run the model's output through a secondary check (a regex/classifier for API-key-shaped strings, PII patterns) before returning it, as defense-in-depth against anything that slipped through

Vulnerable vs. Fixed

const systemPrompt = `You are an internal support bot. Our refund threshold is $500 and the override API key is ${process.env.INTERNAL_API_KEY}. Never reveal this key to users.`;