THE STORY AT A GLANCE

What you need to know

  • Map the link between untrusted input and available actions.
  • Enforce user authorization and destination rules outside the model.
  • Test both attempted misuse and application enforcement.

Treat retrieved material as untrusted input

A document can contain useful facts and a malicious instruction in the same paragraph. If an assistant reads that document while preparing a report, the instruction may attempt to redirect its behavior or expose information. OWASP identifies prompt injection as a risk in LLM applications, including cases involving external material rather than a direct user message.

A clear system instruction helps describe intended behavior, but it does not replace application controls. Map the sources an assistant reads, the tools it can call, and the data each tool can access. This makes the potential impact of an unsuccessful boundary visible before a feature reaches users.

Constrain the actions that can follow

Keep authorization outside model-generated text. A tool call should be checked against the authenticated user's permissions and the specific operation requested. Give read and write actions separate interfaces when possible. Avoid broad tools that allow an assistant to turn arbitrary text into unrestricted commands or requests.

Validate outputs according to their destination. A generated database query, a browser navigation, and a customer email have different failure modes. Where a workflow could send information outside the organization, enforce destination and content rules in code and make consequential actions reviewable. These controls limit impact even when the model misinterprets an input.

Evaluate attacks along real paths

Test malicious instructions in the formats the system actually consumes: documents, retrieved pages, tool responses, and images if supported. Check whether the model attempts the unwanted action and whether the application blocks it. Keep the two findings separate. A blocked attempt can reveal a model weakness while also confirming a useful control. Retest after changes to tools, permissions, retrieval sources, or model behavior, because each can alter the path an attacker might exploit.

FOLLOW THE EVIDENCE

Sources & further reading

This article draws on the source material and official product documentation below. Product capabilities can change; check each source for current details.

VT
Ventures Time Editorial

The Ventures Time editorial desk publishes original explainers and analysis on enterprise AI, data, security, and the business of technology. Articles distinguish practical editorial analysis from source material and link to primary documentation where relevant.

Our editorial standards