Approach to Supervisory Model Development and Validation
The Federal Reserve's supervisory stress test models are developed or selected by Federal Reserve staff and are intended to capture how the balance sheet, risk-weighted assets (RWAs), and net income of firms would be affected by the macroeconomic and financial conditions described in the supervisory scenarios, given the characteristics of their loan and securities portfolios; trading, private equity, and counterparty exposures from derivatives and securities financing transactions (SFTs); business activities; and other relevant factors. In developing supervisory models, Federal Reserve staff draws on economic research and industry practice in modeling these effects on revenues, expenses, and losses. The supervisory models are evaluated by an independent team of Federal Reserve model reviewers.
In February 2019, the Board finalized a Stress Testing Policy Statement that includes modeling principles and policies that guide the development, implementation, validation, and use of supervisory models, after inviting and incorporating comments on these principles and policies from the public.5 Consistent with the principles described in the policy statement, the Federal Reserve designed the system of models to result in projections that are (1) from an independent supervisory perspective; (2) forward-looking; (3) consistent and comparable across covered companies; (4) generated from simple approaches, where appropriate; (5) robust and stable; (6) conservative; and (7) able to capture the effect of economic stress.
The Federal Reserve's models rely on detailed portfolio data provided by firms but generally do not rely on models or estimates provided by firms, consistent with the modeling principle that emphasizes an independent perspective. This framework is unique among regulators in its use of independent estimates of losses and revenues under stress, enables the Federal Reserve to provide the public and firms with credible, independent assessments of each firm's capital adequacy under stress, and helps instill public confidence in the banking system.
The Federal Reserve generally develops its models under an industry-level approach calibrated using data from many financial institutions. This approach reflects modeling principles that favor models resulting in consistent, comparable, and forward-looking projections. The Federal Reserve models the response of specific portfolios and instruments to variations in macroeconomic and financial scenario variables such that differences across firms are driven by differences in firm-specific input data, as opposed to differences in model parameters and specifications. As a result, two firms with the same portfolio receive the same results for that portfolio in the supervisory stress test, facilitating the comparability of results. In addition, the industry-level approach promotes a forward-looking stress test, as it results in models that do not assume that historical patterns will necessarily continue into the future for individual firms. These policies also help to ensure that consistent and comparable supervisory models are forward-looking, robust, and stable.
While the Federal Reserve limits the use of firm-specific fixed effects and the use of dummy variables indicating a loan vintage or specific year, it makes exceptions where appropriate. For example, the Federal Reserve may use firm-specific indicator variables, firm-provided estimates, or third-party models or data in instances in which it is not possible or appropriate to create a supervisory model for use in the stress test, including when supervisory data are insufficient to support an independently modeled estimate of losses or revenues.6 However, the Federal Reserve does not adjust supervisory projections for individual firms or implement firm-specific overlays to model results used in the supervisory stress test. This policy ensures that the supervisory stress test results are determined solely by supervisory models and firm-specific input data.
Policies Related to Model Risk Management, Governance, and Validation
Supervisory model risk management is key to the credibility of the supervisory stress test process. The supervisory model risk management program helps to ensure adherence to consistent development principles, conducts independent model validation, maintains a supervisory model governance structure, and communicates the state of model risk to the members of the Board of Governors on a regular basis. External parties have reviewed several aspects of the Federal Reserve's supervisory stress testing program, including its model risk management framework.
Structure of Model Development and Risk Management Oversight Groups
The Model Oversight Group (MOG), the System Model Validation group, and the Supervisory Stress Test Model Governance Committee are collectively responsible for managing the Federal Reserve's supervisory stress test models and any associated model risks.
The MOG, a national committee of senior staff drawn from across the Federal Reserve System, oversees supervisory model development, implementation, and use. The MOG strives to produce supervisory stress test results that reflect likely outcomes under the supervisory scenarios and ensures that model design across the system of supervisory stress test models results in projections that are consistent with the Federal Reserve's supervisory modeling policies.
The MOG also reviews the results of common model risk management tools7 and assesses potential model limitations and sources of uncertainty surrounding final outputs. A dedicated subgroup of the MOG assists in these efforts and also reviews, assesses, and implements industry standards and best practices for model risk management in stress testing operations. This group is composed of Federal Reserve staff and helps set internal policies, procedures, and standards related to the management of model risk stemming from individual models, as well as the system of supervisory models used to project post-stress capital ratios. In this way, the Federal Reserve's approach reflects the same standards of model risk management that banking organizations are also expected to follow.
Each year, an independent System Model Validation group validates the supervisory stress test models. The System Model Validation group is composed of dedicated full-time staff members not involved in supervisory modeling, supplemented by subject matter experts from across the Federal Reserve System. This group's model validation process includes reviews of model performance, conceptual soundness, and the processes, procedures, and controls used in model development, implementation, and the production of results. For each model, the group annually assesses the model's reliability based on its underlying assumptions, theory, and methods and determines whether any issues require remediation as a result of that assessment. The Model Validation Council, a group of academic experts not affiliated with the Federal Reserve, provides advice to the Federal Reserve on the validation program and activities.8
The MOG and the System Model Validation group are overseen by the director of the Board's Division of Supervision and Regulation. The Supervisory Stress Test Model Governance Committee--a committee of senior Federal Reserve staff that includes representatives from model development, implementation, validation, and scenario design--advises the director on matters related to the governance of supervisory stress test models and facilitates the director's oversight role by providing a regular forum to present and discuss relevant issues. This committee also identifies key model risk issues in the supervisory stress testing program and elevates these issues to the director and the members of the Board of Governors. In 2016, the committee initiated an annual formal communication to the members of the Board of Governors on the structure of the supervisory stress test model risk management program and the state of model risk as determined by each year's model validation process.
External Review of Model Development and Validation Programs
Both internal and external parties have reviewed the development and validation of the supervisory stress test models. In 2015, the Federal Reserve Office of the Inspector General (OIG) reviewed model validation activities and recommended improvements in staffing, model inventories, and communication with management.9 The Federal Reserve has implemented each of the OIG's recommendations, and the OIG has formally closed its findings. Additionally, in 2016, the Government Accountability Office (GAO) issued a report on the Federal Reserve's stress testing and capital planning programs.10 The GAO recognized in its report that the Federal Reserve's stress testing program has played a key role in evaluating and maintaining the stability of the U.S. financial system since the most recent financial crisis. The GAO report included five recommendations as to how the Federal Reserve could improve its model risk management and ensure that its decisions are informed by a comprehensive understanding of model risk. In response, the Federal Reserve has already addressed a number of these recommendations and continues to enhance its stress test model risk management program as consistent with other GAO recommendations.
Data Inputs
The Federal Reserve develops and implements the models with data it collects on regulatory reports as well as proprietary third-party industry data.11
Certain projections rely on aggregate information from the Financial Accounts of the United States (Z.1) statistical release, which is a quarterly publication of national flow of funds, consolidated balance sheet information for each firm, and additional data collected by the Federal Reserve.12
Others rely on the Consolidated Financial Statements for Holding Companies (FR Y-9C) regulatory report, which contains consolidated income statement and balance sheet information for each firm. The FR Y-9C also includes off-balance sheet items and other supporting schedules, such as the components of RWAs and regulatory capital.
Most of the data used in the Federal Reserve's stress test projections are collected through the Capital Assessments and Stress Testing (FR Y-14) information collection, which include a set of annual, quarterly, and monthly schedules (FR Y-14A/Q/M).13 These reports collect detailed data on pre-provision net revenue (PPNR), loans, securities, trading and counterparty risk, losses related to operational-risk events, and business plan changes.
Firms are required to submit detailed loan and securities information for all material portfolios. The definition of materiality is specific to a firm's size and complexity. Portfolio categories are defined in the FR Y-14M and FR Y-14Q instructions. Each firm has the option to either submit or not submit the relevant data schedule for a given portfolio that does not meet the materiality threshold as defined in the instructions. If a firm does not submit data on its immaterial portfolio(s), the Federal Reserve will assign to that portfolio the median loss rate estimated across the set of firms with material portfolios.
While firms are responsible for ensuring the completeness and accuracy of data reported in the FR Y-14 information collection, the Federal Reserve makes efforts to validate firm-reported data and requests resubmissions of data where errors are identified. If data quality remains deficient after resubmission, the Federal Reserve applies conservative assumptions to a particular portfolio or to specific data, depending on the severity of deficiencies. For example, if the Federal Reserve deems the quality of a firm's submitted data too deficient to produce a supervisory model estimate for a particular portfolio, then the Federal Reserve assigns a high loss rate (e.g., 90th percentile) or a conservative PPNR rate (e.g., 10th percentile) to the portfolio balances based on supervisory projections of portfolio losses or PPNR estimated for other firms. If data that are direct inputs to supervisory models are missing or reported erroneously but the problem is isolated in such a way that the existing supervisory framework can still be used, the Federal Reserve assigns a conservative value (e.g., 10th or 90th percentile) to the specific data based on all available data reported by firms. These assumptions are consistent with the Federal Reserve's principle of conservatism and policies on the treatment of immaterial portfolios and missing or erroneous data.
References
5. See 84 Fed. Reg. 6664 (Feb. 28, 2019). Return to text
6. For example, the models to project components of pre-provision net revenue feature firm-specific indicator variables because available data are not sufficiently granular and a firm's own history, after controlling for structural changes over time, is proven to be more predictive of the firm's revenues and expenses under stress than industry-level history. In addition, in order to project trading and counterparty losses, sensitivities to risk factors and other information generated by firms' internal models are used. In cases in which firm-provided or third-party model estimates are used, the Federal Reserve monitors the quality and performance of the estimates through targeted examination, additional data collection, or benchmarking. Return to text
7. Those tools include the use of benchmark models, where applicable, performance testing and monitoring, and sensitivity analysis, which isolates the effect of a change in one model input on the eventual model output. The System Model Validation group examines these tools as part of its process and may recommend modifications to those tools to improve their comprehensiveness. Return to text
8. See "Federal Reserve Board announces the formation of the Model Validation Council," April 20, 2012, https://www.federalreserve.gov/newsevents/pressreleases/bcreg20120420a.htm. Return to text
9. See "The Board Identified Areas of Improvement for Its Supervisory Stress Testing Model Validation Activities, and Opportunities Exist for Further Enhancement," (Washington: Office of Inspector General, October 29, 2015), https://oig.federalreserve.gov/reports/board-supervisory-stress-testing-model-validation-reissue-oct2015.pdf. Return to text
10. See "Additional Actions Could Help Ensure the Achievement of Stress Test Goals," GAO-17-48, (Washington: Government Accountability Office, November 2016), https://www.gao.gov/assets/690/681020.pdf. Return to text
11. In connection with the supervisory stress test, and in addition to the models developed and data collected by federal banking regulators, the Federal Reserve uses proprietary models or data licensed from the following providers: Andrew Davidson & Co., Inc.; ICE Data Services; Bloomberg L.P.; CBRE Econometric Advisors; CoreLogic Inc.; CoStar Group, Inc.; Cox Enterprises, Inc.; Equifax Information Services LLC; Federal Home Loan Mortgage Corporation; Fitch Ratings, Inc.; Haver Analytics; Kenneth French; Black Knight Financial Services; Markit Group Limited; Moody's Analytics, Inc.; Moody's Investors Service, Inc.; Mergent, Inc.; Morningstar, Inc.; MSCI, Inc.; Municipal Securities Rulemaking Board; Real Capital Analytics, Inc.; S&P Global Market Intelligence; and World Bank. In addition, with respect to the global market shock component of the adverse and severely adverse scenarios, the Federal Reserve uses proprietary data licensed from the following providers: Bloomberg L.P.; ICE Data Services; JPMorgan Chase & Co.; Markit Group Limited; and MSCI, Inc. Return to text
12. Financial Accounts of the United States (Z.1) is available on the Federal Reserve website at https://www.federalreserve.gov/releases/z1/. Return to text
13. The FR Y-14 report forms and instructions are available on the Federal Reserve website at https://www.federalreserve.gov/apps/reportforms/default.aspx. Return to text