Incident Patterns in Production LLM Deployments

Key findings

The most striking finding is the concentration of incidents in PSF domains 1 and 2. Input governance failures — unvalidated user inputs reaching the model — and output validation failures — model outputs consumed downstream without schema validation — together account for more than two-thirds of the incidents in this dataset. Both are preventable with straightforward engineering controls.

Incident distribution by PSF domain

The prompt injection problem

Prompt injection — the technique by which adversarial content in user inputs or retrieved documents causes the model to deviate from its intended behaviour — appears in 13 of the 47 incidents (28%). This is consistent with broader industry reporting and represents the fastest-growing incident category in the dataset.

In 9 of the 13 injection incidents, the attack vector was retrieved content: documents, emails, or web pages that the system was asked to summarise or process contained embedded instructions. The remaining 4 involved direct user manipulation of the prompt structure.

What resolves incidents fastest

Across the 47 incidents, time-to-resolution correlated most strongly with two factors: whether the system had observable logging (PSF-4) and whether a human escalation path existed (PSF-6). Systems with both resolved incidents in a median of 4.2 hours. Systems with neither had a median resolution time of 31 hours.

The intervention pattern most associated with fast resolution was the existence of a named incident owner — a person whose responsibility included monitoring the system — combined with an alerting threshold configured before deployment. 38 of the 47 incidents did not have both conditions met at the time of the incident.

Methodology and limitations

Incident data was contributed voluntarily by PAI community members under a standard anonymisation protocol. Contributors provided incident timelines, system descriptions (redacted), root cause assessments, and resolution notes. Classification by PSF domain was performed by two reviewers independently, with disagreements resolved by discussion. The dataset is not a random sample; it reflects the types of incident that community members chose to share and is likely biased toward incidents that were ultimately resolved. Severe or ongoing incidents are probably under-represented.