Production Safety Framework
Eight domains covering the full lifecycle of production AI deployment. Each domain contains five competency statements that form the basis of PAI certification assessment.
Input Governance
Every input reaching an AI model must be validated, sanitised, and treated as untrusted.
All user-supplied input is sanitised before model processing. Injection patterns are detected and handled.
Input schemas are validated: type, length, and structure are checked before the model call.
Inputs are logged with PII redacted. The system maintains an audit-safe record of what the model received.
Rate limiting is applied to all AI endpoints. Abuse patterns trigger alerts, not silent failures.
The system prompt is treated as a security boundary. User content cannot override system-level instructions through normal operation.
Output Validation
Model outputs are untrusted until validated. No output should reach users or downstream systems without a validation layer.
All model outputs pass through a validation layer before being delivered to users or downstream systems.
Output schema is enforced: type, required fields, and length constraints are checked post-generation.
Content filtering is applied to detect harmful, biased, or out-of-scope responses before delivery.
Confidence or uncertainty signals (where available) are logged. Low-confidence outputs are flagged or escalated.
Edge cases and model refusals are handled gracefully. Failures surface as defined error states, not silent gaps.
Data Governance & PII
Personal and sensitive data must be handled with documented policies covering collection, processing, retention, and deletion.
PII in AI inputs and outputs is identified, classified, and handled according to a documented policy.
Inference logs containing PII have defined retention periods that are enforced technically, not just in policy.
Right-to-erasure requests can be fulfilled without requiring full model retraining.
Training and fine-tuning data provenance is documented. Third-party data is licensed for the intended use.
Data residency requirements are understood and implemented. Cross-border transfer risks are assessed.
Model Selection & Evaluation
Model selection must be documented, justified, and subject to ongoing evaluation as models and use cases evolve.
Model selection decisions are documented with a rationale specific to the production use case and risk profile.
Benchmark results relevant to the production use case have been reviewed and documented before deployment.
Model updates and provider changes are subject to a regression testing gate before production promotion.
Fallback behaviour is defined for provider outages. Degraded operation is preferable to silent failure.
Vendor terms and data processing agreements are reviewed and permit the intended use case.
Human Oversight
Autonomy levels must be defined and justified. High-stakes decisions require human review or override mechanisms.
Autonomy level is explicitly defined for each AI decision point: advisory, supervised, or autonomous.
Human review is implemented for all AI decisions that are high-stakes, irreversible, or affect vulnerable individuals.
Users are clearly informed when interacting with an AI system and understand what the system can and cannot do.
Override mechanisms exist for all AI-driven actions. Humans can intervene, correct, or reject AI outputs.
Human oversight design is reviewed after model updates, scope changes, and significant usage pattern shifts.
Observability & Monitoring
Production AI systems must be continuously monitored for performance, quality, and behavioural drift.
Latency, error rate, and throughput are monitored with alerts configured for deviation from baseline.
Model output quality is monitored via sampling or automated evaluation. Quality degradation triggers review.
Usage patterns are monitored for anomalies indicating abuse, adversarial use, or scope creep.
Dashboards provide operations and engineering teams with real-time visibility into AI system health.
Logs are retained for at least 30 days and are structured to support incident investigation.
Incident Response
AI-specific failure modes require a documented response plan with defined escalation paths and recovery procedures.
A documented incident response plan covers AI-specific failure modes including harmful output, data leakage, and model failure.
The team can disable, degrade, or isolate the AI system within 15 minutes of an incident trigger.
Post-incident reviews produce written root cause analysis and remediation tracking within 5 business days.
Customer-facing communication templates for AI incidents are prepared and approved before deployment.
Incident response procedures have been rehearsed through tabletop exercises or documented drills.
Ethics & Accountability
Every production AI system must have a named accountable owner and documented accountability chain.
A named accountable owner is documented for each production AI system. Accountability does not reside in a team, role, or vendor.
Bias and fairness evaluation has been conducted for the use case. Known limitations are documented.
The system complies with applicable AI regulation. Regulatory obligations are tracked as a living document.
Accountability documentation is accessible to internal audit, legal, and regulators on request.
Ethical review has been conducted for sensitive or high-impact use cases. Review is documented and dated.
The Production Safety Framework is published under Creative Commons Attribution 4.0 (CC BY 4.0). You may freely use, share, adapt, and build on this framework for any purpose, including commercially, with attribution to the Production AI Institute. See how to cite the PSF for citation formats.