Prompt Injection: the trick used to manipulate an LLM

It makes mistakes because it doesn't have an engine of truth inside it. It only has a statistical language engine. When it responds, it is not stating what is true: it is completing a sentence in the most probable way based on billions of examples.

An LLM doesn't make mistakes solely because of its own faults. Sometimes it errs because someone deliberately makes it do so. Jailbreaking is not a cyber attack in the usual sense: no one breaks into a server, no one forces a password, no one breaks a digital lock.

Bugs, jailbreaks, prompt injections. Three different problems, one common root: an LLM does not follow rules written in code — it has learned behaviors from billions of examples. This is why fixing it is much more complex than applying a simple patch.