Board of Governors of the Federal Reserve System

Description of the Survey

The Survey of Household Economics and Decisionmaking was fielded from October 11 through November 12, 2018. This is the sixth year of the survey, conducted annually in the fourth quarter of each year since 2013.³¹ Staff of the Federal Reserve Board write the survey questions in consultation with other Federal Reserve System staff, outside academics, and professional survey experts.³²

Ipsos, a private consumer research firm, administers the survey using its KnowledgePanel, a nationally representative probability-based online panel. Ipsos selects respondents for the KnowledgePanel based on address-based sampling (ABS).³³ SHED respondents are then selected from this panel.

Survey Participation

Participation in the 2018 SHED depends on several separate decisions made by respondents. First, they agreed to participate in Ipsos' KnowledgePanel and then they completed an initial demographic profile survey. According to Ipsos, 12.5 percent of individuals contacted to join KnowledgePanel agreed to join (recruitment rate), and 64.2 percent of recruited participants completed the initial profile survey and became a panel member (profile rate). Finally, selected panel members agreed to complete the 2018 SHED.

Of the 21,137 panel members contacted to take the 2018 SHED, 11,440 (excluding breakoffs) participated, yielding a final-stage completion rate of 54.1 percent. All the stages taken together, the cumulative response rate is 4.3 percent. The final sample used in the report includes 11,316 respondents.³⁴

Targeted Outreach and Incentives

To increase survey participation and completion among hard-to-reach demographic groups, Board staff and Ipsos developed a new communication plan and targeted monetary incentives. The target groups—young adults ages 18 to 29, adults with less than a high school degree, and minorities—received frequent email reminders and text messages, as well as increasing monetary incentives. The incentives to take the survey for these groups started at $5 and in some cases increased modestly. Respondents outside the target groups received less frequent communication and a nominal monetary incentive.

Of the nonrespondents in the target groups—slightly more than one-quarter of the survey sample—who were offered an incentive, 14.5 percent took the survey and received the incentive. Half accepted the second offer, while the rest split about evenly between the first and third offers.

Targeted incentives markedly improved the completion rate for the target groups (table 31). More than 53.4 percent of the target groups as a whole completed the survey, up from 43.7 percent achieved in the 2017 survey, a nearly 10 percentage point increase. The increase in completion rates was largest for those with less-than-high-school-degree group (13.5 percentage points) and young adults (12.8 percentage points). The completion rate for minorities increased 6.0 percentage points.

Table 31. Survey completion rate by incentive groups

Altogether, the new communication plan and targeted incentives reduced the differences in response rates across subpopulations and improved the quality of the final data.

Survey Questionnaire

The median time to complete the survey in 2018 was 21 minutes, 3 minutes shorter than the previous survey. The shorter interview length reflects an effort to lessen respondent burden. The number of questions was reduced and the length of the questionnaire was shortened. Working with survey design experts at NORC at the University of Chicago, Board staff also made the question wording clearer to improve comprehension. Most new survey questions went through this technical review, as well as review by subject-matter experts, to minimize potential confusion among respondents.

Because one motivation for the survey is to understand where there may be vulnerabilities or weaknesses in the economy, one priority in selecting questions is to provide information on the financial experiences and challenges among low- and moderate-income populations. The questions are intended to complement and augment the base of knowledge from other data sources, including the Board's Survey of Consumer Finances. In addition, some questions from other surveys are included to allow direct comparisons across datasets.³⁵ The full survey questionnaire can be found in appendix A of the supplemental appendixes to this report (see https://www.federalreserve.gov/consumerscommunities/shed_publications.htm).

Survey Mode

The SHED is administered to respondents entirely online. Online interviews are less costly than telephone or in-person interviewing, and can still be an effective way to interview a representative population.³⁶ Ipsos' online panel offers some additional benefits. Their panel allows the same respondents to be re-interviewed in subsequent surveys with relative ease, as they can be easily contacted for several years.

Furthermore, internet panel surveys have numerous existing data points on respondents from previously administered surveys, including detailed demographic and economic information. This allows for the inclusion of additional information on respondents without increasing respondent burden. The respondent burdens are further reduced by automatically skipping irrelevant questions based on responses to previous answers.

The "digital divide" and other differences in internet usage could bias participation in online surveys, so recruited panel members who do not have a computer or internet access are provided with a laptop and access to the internet to complete the surveys. Even so, individuals who complete an online survey may have greater comfort or familiarity with the internet and technology than the overall adult population.

Sampling and Weighting

The SHED sample is designed to be representative of adults ages 18 and older living in the United States. It includes a main sample and an oversample (table 32) of individuals with a household income less than $40,000 per year ("lower-income oversample"). The completion rate is somewhat lower among the lower-income oversample (48.5 percent) than the main sample (55.4 percent), reflecting the fact that these lower-income adults are harder to reach in surveys.

Table 32. Survey sample and response disposition

The Ipsos methodology for selecting a general population sample from KnowledgePanel ensures that the resulting sample behaves as an equal probability of selection method (EPSEM) sample. This methodology starts by weighting the entire KnowledgePanel to the benchmarks in the latest March supplement of the Current Population Survey along several geo-demographic dimensions. This way, the weighted distribution of the KnowledgePanel matches that of U.S. adults. The geo-demographic dimensions used for weighting the entire KnowledgePanel include gender, age, race, ethnicity, education, census region, household income, homeownership status, and metropolitan area status.

Using the above weights as the measure of size (MOS) for each panel member, in the next step a probability proportional to size (PPS) procedure is used to select study specific samples. Since this survey includes a lower-income oversample, the departures caused by this oversample from an EPSEM design are corrected by adjusting the corresponding design weights accordingly with the Current Population Survey benchmarks serving as reference points.

After the survey collection is complete, statisticians at Ipsos adjust weights in a post-stratification process that corrects for any survey nonresponse as well as any non-coverage or under- and over-sampling in the study design. The following variables were used for the adjustment of weights for this study: age, gender, race, ethnicity, census region, residence in a metropolitan area, education, and household income. Demographic and geographic distributions for the noninstitutionalized, civilian population age 18 and older from the March Current Population Survey are the benchmarks in this adjustment.

Although weights allow the sample population to match the U.S. population (not in the military or in institutions, such as prisons or nursing homes) based on observable characteristics, similar to all survey methods, it remains possible that non-coverage, nonresponse, or occasional disparities among recruited panel members result in differences between the sample population and the U.S. population. For example, address-based sampling likely misses homeless populations, and non-English speakers may not participate in surveys conducted in English.³⁷

Despite an effort to select the 2018 SHED sample such that the unweighted distribution of the sample more closely mirrors that of the U.S. adult population, the result shows that there is room for further improvement. This likely reflects the fact that the distribution of the survey respondents is influenced by the composition of the KnowledgePanel, from which the survey sample is drawn, and is the final step of a multistage process.

 

References

 

 31. Data and reports of survey findings from all past years
are available at https://www.federalreserve.gov/consumerscommunities/shed.htm. Return to text

 32. The survey instrument was also available for public comment through the Federal Reserve Board's website. Return to text

 33. Prior to 2009, respondents were also recruited using random-digit dialing. Return to text

 34. Of the 11,440 respondents who completed the survey, 124 are excluded from the analysis in this report due to either leaving responses to a large number of questions missing, completing the survey too quickly, or both. Return to text

 35. For a comparison of results to select overlapping questions from the SHED and Census Bureau surveys, see Jeff Larrimore, Maximilian Schmeiser, and Sebastian Devlin-Foltz, "Should You Trust Things You Hear Online? Comparing SHED and Census Bureau Survey Results," FEDS Notes (Washington: Board of Governors, October 15, 2015), https://www.federalreserve.gov/econresdata/notes/feds-notes/2015/comparing-shed-and-census-bureau-survey-results-20151015.html. Return to text

 36. See David S. Yeager et al., "Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples," Public Opinion Quarterly 75, no. 4 (2011): 709–47. Return to text

 37. For example, while the survey does weight to match the race and ethnicity of the entire U.S. adult population, there is evidence that the Hispanic population in the survey is somewhat more likely to speak English at home than the overall Hispanic population in the United States. Sixty-five percent of Hispanics who responded to the SHED speak Spanish at home, versus 72 percent of the overall Hispanic population who do so based on the 2017 American Community Survey. See table B16006 at https://factfinder.census.gov. Return to text