Report to the Congress on Credit Scoring and Its Effects on the Availability and Affordability of Credit
Submitted to the Congress pursuant to section 215 of the Fair and Accurate Credit Transactions Act of 2003
August 2007
Print (1.84 MB PDF)
Limitations of the Analysis
Section 215 of the Fact Act asks for four related analyses regarding the use of credit scoring in credit markets. The first is the effect of credit scoring on the availability and affordability of financial products to consumers in general. The second is an analysis of the empirical relationship between credit scores and actual losses experienced by lenders. The third is an evaluation of the effect of scores on the availability and affordability of credit to specific population groups. Finally, the fourth is an evaluation of whether credit scoring in general, and the factors included in credit-scoring models in particular, may result in negative or differential effects on specific subpopulations and, if so, whether such effects could be mitigated by changes in the model development process.
Different approaches were taken to conduct each of these four analyses. The approach used to assess the general effect of credit scoring on the availability and affordability of credit was to rely on evidence from public comments and previous studies on the topic and to obtain indirect evidence from the Survey of Consumer Finances. The ideal way of addressing this question would have been to conduct a "before and after" study of the effects of the introduction of credit scoring on the availability and affordability of credit. Such an endeavor was not possible because credit scoring has been in use for many years, and it is difficult to distinguish the effects of scoring from economic and other changes that took place over the same time period. Also, the available public research is quite limited, perhaps because most analytical studies were proprietary and are not part of the public record. The approach taken here cannot conclusively address these concerns. Thus, our conclusions in this area can only be suggestive.
The approach taken to examine the empirical relationship between credit scores and actual losses experienced by lenders and to examine the effect of scores on the availability and affordability of credit to specific population groups relied on a nationally representative sample of individuals drawn from credit-reporting agency files. There are several limitations to this approach. First, the analysis was limited to credit history scores. Second, the data only included two commercially available credit scores. Third, the definition of performance was dictated by the time periods for which the samples were drawn. The resulting 18-month performance period is on the short end of the timeframes considered by many in the industry. Further, the time period used to evaluate performance represented a relatively favorable period of macroeconomic performance. Consequently, the absolute levels of performance observed here may overstate the performance one would expect in a less favorable economic climate.
The issues of loan performance and the availability and affordability of credit to different populations were addressed using multivariate analyses, which were restricted to information contained in the credit records as supplemented by demographic information from the SSA and data based on location. However, population groups differ widely along many financial and nonfinancial dimensions that are not reflected in credit records, and those other factors may affect credit performance and the conclusions one might draw about differences across populations. So, for example, the overperformance or underperformance of a demographic group may derive from financial or nonfinancial characteristics (such as wealth or employment experience) that bear on performance and that are correlated with the demographic characteristic but are not included in the credit records.
Another issue in this section of the analysis is the fact that performance and loan terms could be ascertained only for individuals receiving credit. It is reasonable to expect that individuals denied credit would have experienced both worse performance and higher interest rates; however, these outcomes are not included in the data as such individuals did not get loans. To the extent that individuals experiencing denials disproportionately have low credit scores, inclusion of these outcomes would likely have made the performance or interest rate curves steeper. The assessment of denial rates using the inquiry proxy is subject to the same limitation. Individuals who know that they have a low credit score, or believe that they do, may act under the assumption that they will be denied credit if they apply for it. If so, they are being "discouraged" from applying for credit, and the observed relationship between credit score and denial rate would then be less steep than it would be if everyone wanting credit applied for it. A final issue in this section is the fact that information on demographic characteristics had to be imputed for a portion of the sample. Tests suggest that the results here are generally robust. However, for some population segments, such as marital status, concerns may still remain.
The fourth analysis was conducted using a credit history scoring model developed by Federal Reserve staff. We attempted to emulate the process used by industry model developers in estimating credit-scoring models. However, our approach was inevitably approximate. For example, data restrictions forced a number of limitations to our approach, and there is no uniform industry methodology. In addition, the fact that industry modelers may have made different decisions or relied upon different samples clearly limits the generalizations that can be made from our results. This would be the case under any circumstances involving the construction of a new model.
Additional concerns are raised about our model development because of the relatively small sample used for estimation. The small sample size prevented evaluation of the FRB base model on an out-of-sample basis (that is, on a sample of individuals different from that used to develop it). Also because of the small sample, the FRB base model was developed with fewer scorecards than are typically used in the industry's credit history scoring models; consequently, the model has fewer credit characteristics than is typical in the industry. Having relatively few scorecards makes it difficult to identify credit characteristics that might have a differential effect on populations that could constitute other possible scorecards.
A limitation that runs through all four of the analyses is the decision to focus on credit history scoring models, as opposed to the broader class of scoring models. Much of the underwriting and pricing of credit relies upon credit-scoring models that incorporate factors not included in the records of the credit-reporting agencies. Further, the underwriting process may use other information that is judgmentally combined with credit scores in making final decisions on underwriting and pricing. The role of some of these other factors could mitigate or alter some of the conclusions reached in the present study.