AI Safety & Governance

6.9.1Definition

AI safety and governance covers the policies, guardrails, and review processes that keep an AI system's behavior within acceptable, intended bounds — refusing harmful requests, avoiding biased or discriminatory output, protecting user data, and remaining accountable when something goes wrong. It applies at both the model level (what the underlying model will and won't do) and the product level (what a specific application built on it allows).

6.9.2Why It Exists

An AI system that can take real actions or influence real decisions (6.5, 2.15) carries real-world consequences when it behaves unexpectedly — leaking data, producing harmful content, or acting outside its intended scope. Safety and governance practices exist because these failure modes are foreseeable categories, not unpredictable edge cases, and addressing them proactively is dramatically cheaper than responding after an incident.

6.9.3Core Governance Areas

• Content and behavior boundaries — clear policies on what the system will refuse, and testing that those boundaries hold under adversarial input, not just typical use.
• Data privacy — what user data is sent to the model, retained, or used for further training, disclosed clearly and handled per relevant regulation.
• Human oversight — meaningful human review or approval for high-stakes AI-driven decisions, rather than full autonomy by default.
• Auditability — logging (5.13) what the system did and why, so an unexpected outcome can be investigated after the fact.

6.9.4Common Mistakes

• No adversarial testing before launch, discovering boundary failures only after a user finds them in production.
• Granting an agent (6.5) full autonomy over high-stakes actions (financial transactions, irreversible deletions) with no human-in-the-loop checkpoint.
• Unclear data handling disclosure, leaving users unaware of what happens to information they share with an AI feature.
• No logging of AI decisions, making it impossible to reconstruct what happened when an unexpected output is reported after the fact.

6.9.5Best Practices

• Test safety boundaries adversarially before launch, not only with typical, well-intentioned inputs.
• Require human confirmation for any AI-initiated action with real-world, hard-to-reverse consequences.
• Log AI decisions and their inputs in enough detail to reconstruct and audit them later.