Insecure Function-Calling / Plugin Design

LLM tool/function-calling lets a model invoke real backend code by producing structured arguments. If those arguments are used the same way any other unvalidated user input would be — an unparameterized SQL fragment, a raw shell command, an unchecked file path — the model becomes an unpredictable new path into every classic injection vulnerability.

The arguments a model produces for a tool call are still attacker-influenceable input (directly, or via prompt injection) — they need exactly the same validation as a value coming from an HTML form

Prefer narrow, strongly-typed tool parameters (an enum of allowed report names) over a single free-text 'query' or 'command' string that gets passed through to something powerful

A tool's docstring/description is read by the model to decide how to use it — an overly permissive description ('runs any SQL query') invites the model to actually do that when asked, even by an injected instruction

Validate tool arguments server-side, in your own code, even though the model API may already validate them against a JSON schema — schema validation checks shape, not safety

Want this flow explained in detail as it plays?

POST/api/agent/run

Headers

Host:app.flowground.dev

Content-Type:application/json

Body

Quick-Insert Payloads

Defense in depth

Design tools with the narrowest possible typed parameters — an enum of pre-approved report names instead of a raw SQL string, a document ID looked up server-side instead of a raw file path
Validate every tool argument against an explicit allowlist/schema in your own backend code before acting on it, exactly as you would for any other externally-supplied input
Give each tool the minimum underlying database/filesystem privilege it needs — a reporting tool's database credential should not have DROP or DELETE privileges at all

Vulnerable vs. Fixed

const runReportTool = {
  name: "run_report",
  description: "Runs any SQL query against the reporting database and returns the results.",
  parameters: { sql: { type: "string" } },
  execute: async ({ sql }) => db.raw(sql), // whatever the model generated, executed verbatim
};