The Federal Reserve Board eagle logo links to home page

Skip to: [SSBF 2003 Home] [Printable Version (PDF)]
2003 SSBF: Methedology Appendix A Screen Reader version

Appendix A

Pretest One Debriefing Memo

A National Organization for Research at the University of Chicago Logo

Memorandum

April 29, 2004

To: John D. Wolken

From: Carol Emmons

Re: 2003 Survey of Small Business Finances

Minutes from Pretest 1 Telephone Interviewer Debriefing

Participants:

Federal Reserve Board: Traci Mach, John Wolken

NORC: Bob Bailey, Mireya Dominguez, Carol Emmons, Terri Kowalczyk, Bill Sherman, NORC Telephone Interviewers, and Phil Panczuk (by phone).

Minutes:

1. Introduction and Ground Rules

Bill Sherman opened the meeting and explained the ground rules. Introductions were accomplished by having each person in the room say their name and tell something interesting about himself or herself or about a recent vacation. Bill emphasized the importance of open and honest participation. He noted that the purpose of the debriefing is to evaluate the survey, not the interviewers.

2. General Observations and Suggested Topics for the Day

Carol Emmons led this portion of the meeting. She explained that the purpose was to identify topics that the interviewers wanted to make sure were discussed during the debriefing. The topics the interviewers identified were the following:

Main Interview:

TNMS Set Up:

Training:

Survey Procedures:

Screener:

3. Building on Strengths: Identifying What Works and Why

Carol Emmons also led this portion of the meeting. The purpose was to identify the things that interviewers found worked well during the first pretest. The interviewers identified the following items:

4. Cooperation, Refusals and Gatekeepers

Bob Bailey led this portion of the meeting. The purpose was to identify the main challenges involved in gaining the cooperation of gatekeepers and respondents with the pretest. Another purpose was to identify and share solutions to overcoming gatekeeper and respondent objections. This session covered a wide range of topics related to gaining cooperation.

Screener vs. Main Interview. In general, the interviewers found it easier to gain cooperation with the main interview than with the screener. It appears that once the owner or proxy agrees completes the screener, he or she is willing to "go the rest of the way," and complete the main interview. Exceptions to this are owners for whom English is not their first language, owners who work directly serving customers, and owners who work in noisy establishments (e.g., auto repair shops).

Interviewers felt, however, that the time between the screener and main interview should be made as short as possible to take advantage of the rapport developed during the screener. Interviewers also felt that some owners were "turned off" by the worksheets because they look complicated and time-consuming to complete.

Survey Introduction. The interviewers all agreed that the screener introduction needed to be changed. They found that simply asking to speak to the owner of the business was a much more effective way to get to speak to the owner, than first introducing themselves and the study to a gatekeeper. The interviewers recommended that we postpone the introduction until we are speaking with the person we want to interview.

There were differences of opinion among the interviewers about whether it was harmful to mention the length of the interview in the introduction. One interviewer noted that respondents sometimes held him to that amount of time. One interviewer said that a strategy he found useful was to ask the respondent for five minutes of his or her time, rather than telling the respondent how much time was needed. Another interviewer thought it better to be vague, i.e., "This will only take a few minutes."

3-attempt Rule to Reach the Owner. Interviewers sometimes found the "3-attempt rule" to reach the owner as a significant challenge during screening. In some instances, it was clear on the first call that the interviewer would never speak to the owner, yet the CATI system required three attempts to reach the owner before it allowed the interviewer to screen a proxy respondent. The interviewers feel that there needs to be a way to circumvent the 3-attempt rule when circumstances demand it.

Proxy Respondents. For some types of small businesses, such as physicians' and dentists' offices, interviewers found that there was no point in asking to speak to the owner, because the physician or dentist would never take the call. One interviewer noted that in such instances, he would say to the gatekeeper, "I want to make sure the owner knows what is going on, so please talk to the owner about this, and I will call back." Interviewers also felt that having the owner's title (i.e., Mr., Ms., Dr.) would allow them to ask for the owner in the proper way, e.g., May I speak with Dr. So-and-so.

One interviewer made the distinction between owner-designated proxies and self-designated proxies. This interviewer thought it would be helpful to have different introductions for different types of proxy respondents.

Reasons for Refusals. Interviewers mentioned the following as the most common reasons for refusing to do the screening interview:

Strategies for Gaining Cooperation. Interviewers mentioned the following strategies for gaining cooperation:

Changes to Help Interviewers Gain Respondent Cooperation. The interviewers identified the following changes they would like to see made, in the questionnaire, call management system, and survey procedures, to help them gain respondent cooperation with the survey:

Collecting Respondent Email Address. When asked how respondents reacted to being asked for their email address, interviewers commented that respondents often asked how this information would be used. Interviewers were confused about how to code a "no" response, i.e., as "no email address," or as a refusal to give the email address.

4. Unusual and Difficult Situations

Mireya Dominquez led this portion of the discussion. The purpose was to identify unusual situations that came up during pretest data collection, that interviewers felt ill-equipped to handle. The following situations were noted:

5. Main Interview Questionnaire

Bill Sherman led the discussion for this topic. The purpose was to identify questions or skip logic in the main interview questionnaire that interviewers found problematic. The interviewers made the following general comments about the main interview questionnaire.

The interviewers also made the following comments about specific sections in the questionnaire:

Section B: Organization Demographics
Question Comment
B3 If the number of owners entered at A10.1 is "2," CATI will not accept "sole proprietorship" as the firm type.

Section C: Personal Characteristics of Owners
Question Comment
C2 When a husband and wife are co-owners, either may be reluctant to name a majority owner. In this situation, some interviewers found it helpful to say, "Let's just start by talking about you."
C30 One interviewer thought this question (which asks if the firm is publicly traded) should be asked earlier because earlier questions sometimes make the answer obvious.
C32 Change QxQ to explain that we are looking to identify the owner who has owned the firm the longest time, and to find out when that owner took ownership.

Section F: Use of Credit and Financing
Question Comment
General Respondents rarely mentioned additional institutions after completing sections E and F.
F32.1 Since vehicle loans are usually collateralized by the vehicle, the vehicle should be the first response option.

Section MRL: Most Recent Loan
Question Comment
General Questions seem redundant with earlier questions in sections E and F.

Section G: Use of Other Financial Services
Question Comment
G11 Respondents confuse bank with the bank holding company that provides them with credit card processing services. Interviewers should be warned about this in training.

Section L: Trade Credit
Question Comment
General Smaller firms are unfamiliar with the term, "trade credit." Most refer to this as "having an account." Should add this and the term "invoices" to QxQ.

Section M: New Equity Investments in Firm
Question Comment
General The QxQs need to be written in shorter sentences.
READ 27 Needs to be shortened.
M1 Suggest re-wording as, "Did someone invest in your company?"

Section P: Income and Expenses
Question Comment
P6 and P8 One interviewer thought that using negative numbers to indicate a loss may be error-prone.

Section R: Assets
Question Comment
R2 Needs to be changed in CATI to match hardcopy version of questionnaire.

7. Screener Issues

This section was led by Terri Kowalczyk. The purpose was to identify problems and issues with the screening questionnaire. The following issues were identified by the interviewers:

8. Institution Look Up

This section was led by Bob Bailey. The purpose was to get interviewer reactions to the institution look-up function in the main interview CATI questionnaire. Interviewers reported that they could usually find the branch the respondent used in the look-up table. The response time for the look up was good. Finally, a surprising number of respondents knew the zip codes of their banks, but less knew the zip codes of non-depository institution sources.

9. Fading Respondents, Regaining Cooperation

Due to earlier sections taking longer than plan, this section was skipped. However, interviewers reported that most respondents were willing to complete the interview, once they started it. (See also module 4, Gaining Cooperation.)

10. Telephone Number Management System (TNMS)

This session was led by Mireya Dominquez. The purpose was to discuss possible improvements to the TNMS to facilitate the interviewers' job. Interviewers made the following suggestions:

11. Job Aids

This discussion was led by Mireya Dominguez. The purpose was to identify necessary improvements to the interviewer Job Aids designed for Pretest 1 as well as to identify additional job aids that interviewers thought would be helpful.

Regarding existing job aids, the interviewers suggested adding the breakpoint function to Job Aid #7: CATI Functions.

They also suggested revising the answering machine script as follows:

Interviewers also suggested developing the following two new job aids for the main study:

Finally, the interviewers commented that the job aids were not very accessible in the binder. Job aids need to be posted at the interviewing station.

12. Contact Materials

Pre-screening Materials. Interviewers reported that these were not very memorable to respondents. They agreed that putting the Federal Reserve Board seal on the envelope would cause respondents to pay more attention to these materials. They also agreed that the advance letters seemed to help if the respondent recalled receiving them. Finally, the interviewers liked the revised version of the project director letter that will be used for Pretest 2.

Worksheet Materials. As described above, the interviewers reported that some respondents were put off by the worksheets because they look long and complicated. Accountants seem to like the worksheets.

13. Incentives

Terri Kowalczyk led this discussion. The purpose was to collect interviewers' impressions of how helpful the incentives were in gaining respondent cooperation.

Some interviewers thought the incentives were very helpful in gaining respondent cooperation; other interviewers felt that the incentives did not make much difference. Nobody thought the incentives were harmful. A few respondents declined the incentive. The $50 appeared to be more popular with respondents than the Dun & Bradstreet Small Business Solutions package. No respondents expressed concern about their names being sold by Dun & Bradstreet. The interviewers agreed that there needs to be rules about who gets the incentive if a proxy completes all or part of the interview.

14. Interviewer Training and Training Materials

Comments and suggestions about interviewer training were made throughout the discussion of other topics. These comments are summarized in this section.

Using the TNMS:

Using CATI

Mock Interviews

Eligibility Criteria

Main Interview

Continuous Training


This version is optimized for use by screen readers. Descriptions for all mathematical expressions are provided in LaTex format. A printable pdf version is available. Return to Text