Prompt Injection

NIST describes prompt injection as an attack that exploits the concatenation of untrusted input with a prompt constructed by a higher-trust party. In a direct prompt injection, the main user of the system provides instructions that are appended to higher-trust instructions such as the model's system prompt. An indirect prompt injection depends on the attacker's ability to modify external sources of information, such as websites or files, that the application feeds into the model's context. A third party mounts it, and the primary user of the model is often the one harmed.

Injected content does not need to be visible or readable to a human, as long as the model parses it. The impact depends on the business context and on how much agency the model has. It can include disclosure of sensitive information, revealing system prompts or details of the AI system's infrastructure, incorrect or biased output, unauthorized access to functions available to the model, and arbitrary commands in connected systems.

OWASP states that, given the stochastic way models work, it is unclear whether fool-proof prevention exists. Its mitigations limit the impact instead. They include constraining model behavior in the system prompt, validating output formats with deterministic code, filtering input and output, restricting the model's privileges to the minimum it needs, requiring human approval for high-risk actions, separating and marking untrusted external content, and regular adversarial testing.

NIST reports that current models remain highly vulnerable to direct prompt injection. Some mitigations therefore work at the system level: deployers design AI systems on the assumption that a model with access to sensitive data, or with the ability to take undesirable actions, may leak that data or take those actions.