Board of Governors of the Federal Reserve System

The Federal Reserve and Office of the Comptroller of the Currency (OCC) are issuing the attached Supervisory Guidance on Model Risk Management, which is intended for use by banking organizations and supervisors as they assess organizations’ management of model risk. This guidance should be applied as appropriate to all banking organizations supervised by the Federal Reserve, taking into account each organization’s size, nature, and complexity, as well as the extent and sophistication of its use of models (as defined and discussed below).

Banking organizations should be attentive to the possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused, and should address those consequences through active model risk management. The attachment to this SR letter describes in more detail the key aspects of an effective model risk management framework, including robust model development, implementation, and use; effective validation; and sound governance, policies, and controls.

Previous publications issued by the Federal Reserve and OCC have addressed the use of models, with particular focus on model validation.¹ Based on supervisory and industry experience over the past several years, this document expands upon existing guidance—most importantly by broadening the scope to include other key aspects of model risk management.

For the purposes of this document, the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. Models meeting this definition might be used for analyzing business strategies, informing business decisions, identifying and measuring risks, valuing exposures, instruments or positions, conducting stress testing, assessing adequacy of capital, managing client assets, measuring compliance with internal limits, maintaining the formal control apparatus of the bank, or meeting financial or regulatory reporting requirements and issuing public disclosures. The definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgment, provided that the output is quantitative in nature.²

The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. Model risk can lead to financial loss, poor business and strategic decision-making, or damage to a banking organization’s reputation. Model risk occurs primarily for two reasons:  (1) a model may have fundamental errors and produce inaccurate outputs when viewed against its design objective and intended business uses; (2) a model may be used incorrectly or inappropriately or there may be a misunderstanding about its limitations and assumptions. Model risk increases with greater model complexity, higher uncertainty about inputs and assumptions, broader extent of use, and larger potential impact.  Banking organizations should manage model risk both from individual models and in the aggregate.

A guiding principle throughout the guidance is that managing model risk involves "effective challenge" of models:  critical analysis by objective, informed parties that can identify model limitations and produce appropriate changes. Effective challenge depends on a combination of incentives, competence, and influence.

As is generally the case with other risks, materiality is an important consideration in model risk management. If at some banks the use of models is less pervasive and has less impact on their financial condition, then those banks may not need as complex an approach to model risk management in order to meet supervisory expectations. However, where models and model output have a material impact on business decisions, including decisions related to risk management and capital and liquidity planning, and where model failure would have a particularly harmful impact on a bank’s financial condition, a bank’s model risk management framework should be more extensive and rigorous.

Model development relies heavily on the experience and judgment of developers, and model risk management should include disciplined model development and implementation processes that are consistent with the situation and goals of the model user and with the banking organization’s policy. A sound development process includes:  a clear statement of purpose to ensure that the model is developed in line with its intended use; sound design, theory, and logic underlying the model; robust model methodologies and processing components; rigorous assessment of data quality and relevance; and appropriate documentation. An integral part of model development is testing, in which the various components of a model and its overall functioning are evaluated to show the model is performing as intended; to demonstrate that it is accurate, robust, and stable; and to evaluate its limitations and assumptions. Importantly, organizations should ensure that the development of the more judgmental and qualitative aspects of their models is also sound.

All models have some degree of uncertainty and inaccuracy because they are by definition imperfect representations of reality. An important outcome of effective model development, implementation, and use is a banking organization’s demonstrated understanding of and accounting for such uncertainty. Accounting for model uncertainty can include applying well-supported, judgmental, “conservative” adjustments to model output, placing less emphasis on a model’s output, or ensuring that a model is only used when supplemented by other models or approaches.³

Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. Effective validation helps to ensure that models are sound, identifying potential limitations and assumptions and assessing their possible impact. All model components—inputs, processing, outputs, and reports—should be subject to validation; this applies equally to models developed in-house and to those purchased from or developed by vendors or consultants.

Validation involves a degree of independence from model development and use. Generally, validation is done by staff who are not responsible for model development or use and do not have a stake in whether a model is determined to be valid. As a practical matter, some validation work may be most effectively done by model developers and users; it is essential, however, that such validation work be subject to critical review by an independent party, who should conduct additional activities to ensure proper validation. Overall, the quality of the validation process is indicated by critical review by objective, knowledgeable parties and the actions taken to address issues identified by those parties.

Validation activities should continue on an ongoing basis after a model goes into use to track known model limitations and to identify any new ones. Validation is an important check during periods of benign economic and financial conditions, when estimates of risk and potential loss can become overly optimistic and the data at hand may not fully reflect more stressed conditions. Banking organizations should conduct a periodic review—at least annually but more frequently if warranted—of each model to determine whether it is working as intended and if the existing validation activities are sufficient. Key elements of comprehensive validation include:

• Evaluation of Conceptual Soundness. This element involves assessing the quality of the model design and construction, as well as review of documentation and empirical evidence supporting the methods used and variables selected for the model. This step in validation should ensure that judgment exercised in model design and construction is well informed, carefully considered, and consistent with published research and with sound industry practice.
• Ongoing Monitoring. This step in validation is done to confirm that the model is appropriately implemented and is being used and performing as intended.  It is essential to evaluate whether changes in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model and to verify that any extension of the model beyond its original scope is valid. Benchmarking can be used in this step to compare a given model’s inputs and outputs to estimates from alternatives.
• Outcomes Analysis. This step involves comparing model outputs to corresponding actual outcomes. Back-testing is one form of outcomes analysis that involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development at a frequency that matches the model’s forecast horizon or performance window.

The results of the three core elements of the validation process may reveal significant errors or inaccuracies in model development or outcomes that consistently fall outside the banking organization’s predetermined thresholds of acceptability. In such cases, model adjustment, recalibration, or redevelopment is warranted. At times, banking organizations may have a limited ability to use key model validation tools for various reasons, such as lack of data or of price observability. In those cases, even more attention should be paid to the model’s limitations when considering the appropriateness of model usage, and senior management should be fully informed of those limitations when using the models for decision-making. Generally, senior management should ensure that appropriate mitigating steps are taken in light of identified model limitations, which can include adjustments to model output, restrictions on model use, reliance on other models or approaches, or other compensating controls

Developing and maintaining strong governance over the model risk management framework is fundamentally important to its effectiveness. Strong governance provides explicit support and structure to risk management functions through policies defining relevant risk management activities, procedures that implement those policies, allocation of resources, and mechanisms for testing that policies and procedures are being carried out as specified. Strong governance also includes documentation of model development and validation that is sufficiently detailed to allow parties unfamiliar with a model to understand how the model operates, as well as its limitations and key assumptions.

Model risk governance is provided at the highest level by the board of directors and senior management when they establish an organization-wide approach to model risk management. Board members should ensure that the level of model risk is within their tolerance. A banking organization’s internal audit function should assess the overall effectiveness of the model risk management framework, including the framework’s ability to address both types of model risk for individual models and in the aggregate. Whenever a banking organization uses external resources for model risk management, the organization should specify the activities to be conducted in a clearly written and agreed-upon scope of work, and those activities should be conducted in accordance with this guidance. Also, organizations should maintain an inventory of models implemented for use, under development for implementation, or recently retired.

All banking organizations should ensure that their internal policies and procedures are consistent with the risk management principles and supervisory expectations contained in this guidance.

For questions regarding this guidance, please contact David Palmer, Senior Supervisory Financial Analyst, Risk, at (202) 452-2904; Dwight Smith, Senior Supervisory Financial Analyst, Capital & Regulatory Policy, at (202) 452-2773; or Anna Lee Hewko, Assistant Director, at (202) 530-6260. In addition, questions may be sent via the Board’s public website.⁴

Notes: