Approach to Supervisory Model Development and Validation
The Federal Reserve's supervisory stress test models are developed or selected by Federal Reserve staff and are intended to capture how firms' net income and other components of regulatory capital would be affected by the macroeconomic and financial conditions described in the supervisory scenarios, given the characteristics of their loan and securities portfolios; trading and private equity exposures and counterparty exposures from derivatives and securities financing transactions (SFTs); business activities; and other relevant factors. In developing supervisory models, Federal Reserve staff draws on economic research and industry practice in modeling these effects on revenues, expenses, and losses. The supervisory models are evaluated by an independent team of Federal Reserve model reviewers.
In February 2019, the Board finalized a Stress Testing Policy Statement that includes modeling principles and policies that guide the development, implementation, validation, and use of supervisory models, after inviting and incorporating comments on these principles and policies from the public.5 Consistent with the principles described in the policy statement, the Federal Reserve designed the system of models to result in projections that are (1) from an independent supervisory perspective; (2) forward-looking; (3) consistent and comparable across firms; (4) generated from simple approaches, where appropriate; (5) robust and stable; (6) conservative; and (7) able to capture the effect of economic stress.
The Federal Reserve's models rely on detailed portfolio data provided by firms but generally do not rely on models or estimates provided by firms, consistent with the modeling principle that emphasizes an independent perspective. This framework is unique among regulators in its use of independent estimates of losses and revenues under stress, enables the Federal Reserve to provide the public and firms with credible, independent assessments of each firm's capital adequacy under stress, and helps instill public confidence in the banking system.
The Federal Reserve generally develops its models under an industry-level approach calibrated using data from many financial institutions. This approach reflects modeling principles that favor models resulting in consistent, comparable, and forward-looking projections. The Federal Reserve models the response of specific portfolios and instruments to variations in macroeconomic and financial scenario variables such that differences across firms are driven by differences in firm-specific input data, as opposed to differences in model parameters and specifications. As a result, two firms with the same portfolio receive the same results for that portfolio in the supervisory stress test, facilitating the comparability of results. In addition, the industry-level approach promotes a forward-looking stress test, as it results in models that do not assume that historical patterns will necessarily continue into the future for individual firms. These policies also help to ensure that consistent and comparable supervisory models are forward-looking, robust, and stable.
In general, the Federal Reserve only employs firm-specific fixed effects and vintage indicator variables to account for significant structural market shifts or other unusual factors. For example, the Federal Reserve may use firm-specific indicator variables, firm-provided estimates, or third-party models or data in instances in which it is not possible or appropriate to create a supervisory model for use in the stress test, including when supervisory data are insufficient to support an independently modeled estimate of losses or revenues.6 However, the Federal Reserve does not adjust supervisory projections for individual firms or implement firm-specific overlays to model results used in the supervisory stress test. This policy ensures that the supervisory stress test results are determined solely by supervisory models and firm-specific input data.
Policies Related to Model Risk Management, Governance, and Validation
Effective model risk management practices are key to the credibility of the supervisory stress test process. The Federal Reserve's supervisory stress test model risk management program has a governance structure that ensures adherence to consistent development principles, conducts independent model validation, and communicates the state of model risk to the members of the Board of Governors regularly. External parties have reviewed several aspects of the Federal Reserve's supervisory stress testing program, including its model risk management framework.
Structure of Model Development and Risk Management Oversight Groups
The Model Oversight Group (MOG), the System Model Validation (SMV) group, and the Supervisory Stress Test Model Governance Committee (SSTMGC) are collectively responsible for managing the Federal Reserve's supervisory stress test models and any associated model risks.
The MOG and the SMV group are overseen by the director of the Board's Division of Supervision and Regulation. The SSTMGC advises the director on matters related to the governance of supervisory stress test models and facilitates the director's oversight role by providing a regular forum to present and discuss relevant issues.
The MOG, a national committee of senior staff drawn from across the Federal Reserve System, oversees supervisory model development, implementation, and use. The MOG strives to produce supervisory stress test results that reflect likely outcomes under the supervisory scenarios and ensures that model design across the system of supervisory stress test models results in projections that are consistent with the Federal Reserve's supervisory modeling policies.
The MOG also reviews the results of common model risk management tools7 and assesses potential model limitations and sources of uncertainty surrounding final outputs. A dedicated subgroup of the MOG assists in these efforts to meet industry standards and best practices for model risk management in stress testing operations. This group is composed of Federal Reserve staff and helps set internal policies, procedures, and standards related to the management of model risk stemming from individual models, as well as the system of supervisory models used to project post-stress capital ratios. In this way, the Federal Reserve's approach reflects the same standards of model risk management that banking organizations are also expected to follow.
Each year, the independent SMV group validates the supervisory stress test models. The SMV group is composed of dedicated full-time staff not involved in supervisory modeling. The SMV group's model validation process includes reviews of model performance, conceptual soundness, and the processes, procedures, and controls used in model development, implementation, and results production. For each model, the group annually assesses the model's reliability based on its underlying assumptions, theory, and methods and determines whether any issues require remediation as a result of that assessment. Separately, the Model Validation Council, a group of academic experts not affiliated with the Federal Reserve, provides advice to the Federal Reserve on the validation program and activities.8
Providing high-level governance of the supervisory stress test modeling program, the SSTMGC consists of senior representatives from the stress testing model development, model validation, and scenario design functions. The SSTMGC advises the Director of Supervision and Regulation on three core mandates for the supervisory stress test modeling program: providing oversight of model risk management policies, procedures, and activities; ensuring appropriate model risk management governance; and maintaining suitable information flows. The committee produces an annual formal communication to the members of the Board of Governors on the structure of the supervisory stress test model risk management program and the state of model risk as determined by each year's model validation process.
External Review of Model Development and Validation Programs
Both internal and external parties have reviewed the development and validation of the supervisory stress test models. In 2015, the Federal Reserve Office of the Inspector General (OIG) reviewed model validation activities and recommended improvements in staffing, model inventories, and communication with management.9 The Federal Reserve has implemented each of the OIG's recommendations, and the OIG has formally closed its findings. Additionally, in 2016, the Government Accountability Office (GAO) issued a report on the Federal Reserve's stress testing and capital planning programs.10 The GAO recognized in its report that the Federal Reserve's stress testing program has played a key role in evaluating and maintaining the stability of the U.S. financial system since the 2007–09 financial crisis. The GAO report included five recommendations specific to how the Federal Reserve could improve its model risk management and ensure that its decisions are informed by a comprehensive understanding of model risk. In response, the Federal Reserve comprehensively addressed three of these recommendations, which the GAO subsequently closed, and continues to enhance its stress test model risk management practices to address the remaining two GAO recommendations.
Box 1. Methodology Input from the Requests for Reconsideration of Stress Capital Buffer Requirements
The Board's capital plan rule permits a firm to request reconsideration of its determined stress capital buffer requirement (reconsideration process).1 The Federal Reserve is reviewing the input gained from the reconsideration process and will use it to improve its stress testing methodology.
Last year, five firms requested reconsideration of their stress capital buffer requirements, some of which raised issues regarding model methodology. A group of experts within the Federal Reserve System, who are independent of the staff who developed the models, analyzed the arguments made by each firm in favor of reconsideration of its stress capital buffer requirement. Following this process, the Board affirmed the stress capital buffer requirements of these firms.2
As part of their review, however, the independent experts made certain recommendations to the Board towards potential improvements to the stress test models related to issues raised in the reconsideration process. In turn, the Board directed staff to investigate and address, as appropriate, issues relating to interest rate hedges, loss-sharing agreements in certain consumer portfolios, and loans that use fair-value option accounting. Staff are considering these issues and are designing potential next steps in accordance with the Federal Reserve's model development principles and policies.3
1. See 12 C.F.R. § 225.8(h)(2)(i) (2021). Return to text
2. See Board of Governors of the Federal Reserve System, "Federal Reserve Board Announces Individual Large Bank Capital Requirements, which Will Be Effective on October 1," press release, August 10, 2020, https://www.federalreserve.gov/newsevents/pressreleases/bcreg20200810a.htm. Return to text
3. For example, the Federal Reserve has instituted a policy of not using additional input data submitted by one or some of the firms unless comparable data can be collected from all the firms that have material exposure in a given area. Return to text
Data Inputs
The Federal Reserve develops and implements the models with data it collects on regulatory reports as well as proprietary third-party industry data.11
Certain projections rely on the Consolidated Financial Statements for Holding Companies (FR Y-9C) regulatory report, which contains consolidated income statement and balance sheet information for each firm. The FR Y-9C also includes off-balance sheet items and other supporting schedules, such as the components of risk-weighted assets (RWAs) and regulatory capital.
- Most of the data used in the Federal Reserve's stress test projections are collected through the Capital Assessments and Stress Testing (FR Y-14) regulatory report, which includes a set of annual, quarterly, and monthly schedules (FR Y-14A/Q/M).12 The annual FR Y-14A collects quantitative projections of balance sheet, income, losses, and capital across a range of macroeconomic scenarios and qualitative information on methodologies used to develop internal projections of capital across scenarios.
- The quarterly FR Y-14Q collects granular data on various asset classes, including loans, securities, trading assets, and pre-provision net revenue (PPNR) for the reporting period.
- The monthly FR Y-14M is comprised of three retail portfolio- and loan-level schedules, and one detailed address-matching schedule to supplement two of the portfolio and loan-level schedules.
Firms are required to submit detailed loan and securities information for all material portfolios. The definition of materiality is based on a firm's size and complexity.13 Portfolio categories are defined in the FR Y-14M and FR Y-14Q instructions. Each firm has the option to either submit or not submit the relevant data schedule for a given portfolio that does not meet the materiality threshold as defined in the instructions. If a firm does not submit data on its immaterial portfolio(s), the Federal Reserve will assign to that portfolio the median loss rate estimated across the set of firms with material portfolios.
While firms are responsible for ensuring the completeness and accuracy of data reported in the FR Y-14 information collection, the Federal Reserve makes efforts to validate firm-reported data and requests resubmissions of data where errors are identified. If data quality remains deficient after resubmission, the Federal Reserve applies conservative assumptions to a particular portfolio or to specific data, depending on the severity of deficiencies. For example, if the Federal Reserve deems the quality of a firm's submitted data too deficient to produce a supervisory model estimate for a particular portfolio, then the Federal Reserve assigns a high loss rate (e.g., 90th percentile) or a conservative PPNR rate (e.g., 10th percentile) to the portfolio balances based on supervisory projections of portfolio losses or PPNR estimated for other firms. If data that are direct inputs to supervisory models are missing or reported erroneously but the problem is isolated in such a way that the existing supervisory framework can still be used, the Federal Reserve assigns a conservative value (e.g., 10th or 90th percentile) to the specific data based on all available data reported by firms. These assumptions are consistent with the Federal Reserve's principle of conservatism and policies on the treatment of immaterial portfolios and missing or erroneous data.
Final stress test results will generally be based on data filed by established regulatory reporting deadlines. The Board will consider revised data where applicable but revised data submitted more than seven weeks after the due date of capital plans will be used at the Board's discretion.
References
5. See 84 Fed. Reg. 6664 (February 28, 2019). Return to text
6. For example, the models to project components of pre-provision net revenue (PPNR) feature firm-specific indicator variables because available data are not sufficiently granular and a firm's own history, after controlling for structural changes over time, is proven to be more predictive of the firm's revenues and expenses under stress than industry-level history. In addition, in order to project trading and counterparty losses, sensitivities to risk factors and other information generated by firms' internal models are used. In cases in which firm-provided or third-party model estimates are used, the Federal Reserve monitors the quality and performance of the estimates through targeted examination, additional data collection, or benchmarking. Return to text
7. Those tools include the use of benchmark models, where applicable, performance testing and monitoring, and sensitivity analysis, which isolates the effect of a change in one model input on the eventual model output. The SMV group examines tools as part of its process with a focus on performance testing and ongoing monitoring and may recommend modifications to those tools to improve their comprehensiveness. Return to text
8. See Board of Governors of the Federal Reserve System, "Federal Reserve Board Announces the Formation of the Model Validation Council," press release, April 20, 2012, https://www.federalreserve.gov/newsevents/pressreleases/bcreg20120420a.htm. Return to text
9. See Board of Governors of the Federal Reserve System and Consumer Financial Protection Bureau, Office of Inspector General, The Board Identified Areas of Improvement for Its Supervisory Stress Testing Model Validation Activities, and Opportunities Exist for Further Enhancement, Evaluation Report 2015-SR-B-018 (Washington: Board of Governors and CFPB, OIG, October 2015), https://oig.federalreserve.gov/reports/board-supervisory-stress-testing-model-validation-reissue-oct2015.pdf. Return to text
10. See Government Accountability Office, "Additional Actions Could Help Ensure the Achievement of Stress Test Goals," (Washington: GAO, November 2016), https://www.gao.gov/assets/690/681020.pdf. Return to text
11. In connection with the supervisory stress test, and in addition to the models developed and data collected by federal banking regulators, the Federal Reserve uses proprietary models or data licensed from the following providers:
Andrew Davidson & Co., Inc.; Black Knight Financial Services; Bloomberg Finance LP; CBRE Econometric Advisors; CoreLogic Inc.; Cox Enterprises, Inc.; Equifax Information Services LLC; Federal Home Loan Mortgage Corporation; Haver Analytics; ICE Data Services; IHS Markit Ltd.; Mergent, Inc.; Moody's Analytics, Inc.; Moody's Investors Service, Inc.; Morningstar, Inc.; Municipal Securities Rulemaking Board; Real Capital Analytics, Inc.; Refinitiv; RiskMetrics Solutions, LLC; S&P Global; and The World Bank.
In addition, with respect to the global market shock component of the severely adverse scenario, the Federal Reserve uses proprietary data licensed from the following providers: Bloomberg Finance LP; ICE Data Services; IHS Markit Ltd; JPMorgan Chase & Co.; Moody's Analytics, Inc.; and RiskMetrics Solutions, LLC. Notes regarding scenario variable data can be found in Board of Governors of the Federal Reserve System, 2021 Stress Test Scenarios(Washington: Board of Governors, February 2021), 15–16, https://www.federalreserve.gov/newsevents/pressreleases/files/bcreg20210212a1.pdf. Return to text
12. The FR Y-14 report forms and instructions are available on the Board's website at https://www.federalreserve.gov/apps/reportforms/default.aspx. Return to text
13. Specifically, the definition of a material portfolio varies depending upon a firm's categorization in the risk-based category framework adopted by the Board for determining prudential standards. See 12 C.F.R. § 252.5 for more information on this framework. Return to text