Model risk management is the practice of identifying, measuring, and controlling the risk that a model produces incorrect, biased, or misapplied output. Codified for US banks in Federal Reserve guidance SR 11-7, the discipline now has to cover AI systems whose behavior depends less on model weights than on the data, permissions, and prompts feeding them.
That shift is the entire problem. A validation program built to interrogate a credit scoring model has very little to say about a large language model answering a regulator's question out of a document repository that changed this morning. The controls are not wrong. They are pointed at the wrong layer.
| DIMENSION | TRADITIONAL MODEL RISK MANAGEMENT | AI MODEL RISK MANAGEMENT |
|---|---|---|
| Primary risk source | Model specification and assumptions | Retrieved content, permissions, and prompts |
| Validation cadence | Periodic, often annual | Continuous, evaluated per answer |
| Inputs | Curated structured datasets | Unstructured documents that change daily |
| Evidence of correctness | Backtesting and benchmark reports | Per-answer provenance and audit trail |
| Typical failure mode | Gradual drift in accuracy | Confident, well-formatted wrong answers |
| Ownership | Model risk team | Model risk, data owners, and security jointly |
What Model Risk Management Means When the Model Is an LLM
Traditional model risk management assumes a bounded artifact. You can read the specification, reproduce the output, and test it against held-out data. Generative AI breaks that assumption in three places at once: the model is usually third party and opaque, the inputs are unstructured and change daily, and the same prompt can return different text on different days.
The practical consequence is that validating the model tells you almost nothing about the risk of the system. An organization can license a well benchmarked frontier model and still ship answers that cite a policy retired eighteen months ago, quote a contract clause the reader was never cleared to see, or assert a certification the company does not hold. None of those failures originate in the model. All of them are model risk in the supervisory sense, because they are incorrect output from an automated system that someone relied on to make a decision.
Why SR 11-7 Still Sets the Standard for Model Risk Management
SR 11-7 defines model risk as arising from two sources: fundamental errors in the model itself, and a sound model being used incorrectly or in the wrong context. That second source has aged unusually well. Most AI incidents in regulated firms are not mathematical failures. They are context failures, where a capable system was pointed at stale, unauthorized, or unapproved material and answered anyway.
The guidance also insists that model risk is managed rather than eliminated, through an ongoing program of disciplined development, independent validation, and governance with named accountability. Read against generative AI, that is less a piece of legacy banking regulation than a specification most firms have not yet implemented.
The Three Lines of Defense Still Apply
The structure survives the technology change. The first line owns the AI system and the content it draws on. The second line validates independently, which now means testing retrieval accuracy and permission enforcement, not only output quality. The third line audits, which requires records that already exist rather than records someone reconstructs after the fact. AI does not need a new governance vocabulary. It needs the existing one enforced inside the system instead of described in a policy document.
Where a Traditional Model Risk Management Framework Breaks
The Model Is No Longer the Main Variable
In a retrieval based AI system, the model composes the sentence and the knowledge layer decides whether the sentence is true. Swap the model and most answers stay roughly the same. Swap the underlying source document and every answer changes. A framework that spends its validation budget on model benchmarks and nothing on source control is measuring the smaller risk. This is also why a model agnostic architecture is a risk control and not merely a procurement preference: when governance lives in the knowledge layer, changing models does not restart validation from zero.
Annual Validation Cannot Cover a Daily Knowledge Base
Periodic validation was reasonable when the inputs were a quarterly data feed. It is not reasonable when the inputs are thousands of documents that different teams edit continuously. A single superseded source creates a staleness cascade: every answer built on it inherits the error, and nothing in the output signals a problem. Continuous validation means each answer carries a record of which source version produced it, so a retired document can be traced forward to everything it touched.
Permissions Are Now a Model Risk Control
Access control used to sit with security, comfortably outside the model risk perimeter. In an AI system that boundary disappears, because a model with broader retrieval rights than its user is a disclosure event waiting for the right question. Attribute based access control at the AI layer, evaluated per query rather than per repository, keeps entitlement enforcement inside the answer path. Sovrinty's approach to zero exfiltration and ABAC treats that boundary as part of the model risk surface, not adjacent to it.
Building a Model Risk Management Framework for AI
1. Inventory AI systems, not just models. The unit of registration is the system: model, retrieval layer, source corpus, permission policy, and use case. An inventory that lists a vendor hosted LLM and stops there is not an inventory.
2. Tier by consequence. An internal drafting assistant and a system producing statements to regulators, customers, or counterparties do not warrant the same scrutiny. Tier on the consequence of a wrong answer, then set validation depth to match.
3. Govern the sources before the model. Every source document needs a named owner, a review date, and an approval state. Unapproved and expired material should be structurally unreachable by the AI, not merely discouraged in a guideline.
4. Validate retrieval and entitlement, not only output. Independent validation should test whether the system retrieves the current approved source and whether it respects the requesting user's permissions. Both are testable, repeatable, and far more diagnostic than a fluency score.
5. Make the record automatic. If producing evidence for an examiner requires a project, the control does not really exist. The audit trail has to be a byproduct of answering, captured per answer at the moment of generation.
Evidence Is the Deliverable
Regulators are converging on the same expectation from different directions. The NIST AI Risk Management Framework organizes its functions around documented, testable governance. The EU AI Act attaches record keeping and traceability obligations to high risk systems, with penalties reaching EUR 35 million or 7 percent of global turnover. Gartner has forecast that 60 percent of enterprise AI projects will be abandoned through 2026 because the data behind them is not AI ready. Different regimes, one requirement: show your work.
That is what Sovrinty is built to produce. The Golden Spike creates an immutable per answer audit trail linking every response to the exact approved source it came from, verbatim enforcement keeps controlled language intact rather than paraphrased into a new claim, and bring your own model support keeps the governance layer constant while the model underneath changes. For financial services teams already running an SR 11-7 program, this is less a new framework than the missing instrumentation for the one they have.
Extending model risk management to AI is mostly a question of where the controls live. If they live in a policy document, validation stays manual and evidence stays retrospective. If they live in the knowledge layer, every answer arrives with its own proof. See how Sovrinty makes that record automatic.
FAQ
Common questions
What is model risk management?
Model risk management is the practice of identifying, measuring, monitoring, and controlling the risk that a model produces incorrect output or is used in a context it was never validated for. It covers the full lifecycle: development, independent validation, ongoing monitoring, and governance with named accountability.
Does SR 11-7 apply to AI and large language models?
Yes for supervised US banking institutions. SR 11-7 defines a model broadly as any quantitative method that processes input into estimates, which captures AI systems used to produce decisions or statements. Most examiners now expect AI use cases to appear in the model inventory and be validated accordingly.
How is AI model risk management different from traditional model risk management?
The main difference is where the risk sits. Traditional model risk concentrates in the model specification, while AI risk concentrates in the retrieved content, the permissions applied, and the prompt. That moves validation toward source control, entitlement testing, and per answer evidence rather than periodic benchmarking alone.
Who owns model risk management for AI systems?
Ownership is shared, which is why it often fails. The business owns the use case, model risk owns validation, data owners own source accuracy and approval state, and security owns entitlements. The three lines of defense structure still works, but each line needs explicit responsibility for the knowledge layer as well as the model.
What should an AI model risk management framework include?
At minimum: a system level inventory, risk tiering by consequence, governed and versioned sources with named owners, independent validation of retrieval and permissions, continuous monitoring, and an automatic per answer audit trail. The last item is what turns the framework into evidence an examiner can review.
How often should AI models be validated?
Formal revalidation should stay periodic and risk tiered, but for AI systems it is not sufficient on its own. Because the source corpus changes daily, monitoring has to run continuously, with material source or model changes triggering revalidation rather than waiting for the annual cycle.
