Appendix A
Pretest One Debriefing Memo
Memorandum
April 29, 2004
To: John D. Wolken
From: Carol Emmons
Re: 2003 Survey of Small Business Finances
Minutes from Pretest 1 Telephone Interviewer Debriefing
Participants:
Federal Reserve Board: Traci Mach, John Wolken
NORC: Bob Bailey, Mireya Dominguez, Carol Emmons, Terri Kowalczyk, Bill Sherman, NORC Telephone Interviewers, and Phil Panczuk (by phone).
Minutes:
1. Introduction and Ground Rules
Bill Sherman opened the meeting and explained the ground rules. Introductions were accomplished by having each person in the room say their name and tell something interesting about himself or herself or about a recent vacation. Bill emphasized the importance of open and honest participation. He noted that the purpose of the debriefing is to evaluate the survey, not the interviewers.
2. General Observations and Suggested Topics for the Day
Carol Emmons led this portion of the meeting. She explained that the purpose was to identify topics that the interviewers wanted to make sure were discussed during the debriefing. The topics the interviewers identified were the following:
Main Interview:
- Proxies who are accountants: Questionnaire is overscripted for these types of proxy respondents, who understand the terminology and do not need long definitions.
- Repetition of firm name and fiscal year ending date: These terms are overused in the questionnaire and make respondents impatient.
- Similar questions: Asking questions that seem similar to ones already asked is annoying to respondents.
- Distance Questions: Need coding instruction for distances less than one mile; need to fix CATI to allow for long distances (screen field too small).
- Closing Script: Need to change to refer to only the documents that R used during the interview. Respondents think we are asking them to send their bank statements, and respond negatively.
- QxQs: Solid blocks of text difficult to read. Need to be reformatted with shorter sentences and more white space. Key phrases should be highlighted.
- TNMS Dispositions: The TNMS does not provide the appropriate dispositions codes at the appropriate stages of the interview process.
- Identity of Screener Respondent: It would be helpful to know who the screener respondent was when calling to conduct the main interview.
- Company Names: Abbreviated company names are hard to decipher.
- Owner Title or Gender: Sometimes the owner's name contained only an initial. Not clear whether to ask for "Mr." or "Ms." Should provide owner title or gender.
- Identity of Earlier Respondent in the Case of a Breakoff: The way the system is currently set up, the interviewer who is calling the case to complete the interview after a breakoff does not know the identify of the respondent interviewed up to that point, unless it is documented in the call notes.
- Frequency of Callbacks: Calling rules need to be reviewed to prevent too-frequent callbacks.
Training:
- Call Notes: Stress the importance, during training, of writing detailed call notes. Stress the importance of recording gatekeeper and proxy names in the call notes.
- Properly Exiting a Case: More emphasis needed in training on how to properly exit a case.
Survey Procedures:
- Answering Machine Script: Change to include "...or to schedule an appointment for your interview." Also mention importance of survey, and the fact that we are not selling anything
- Time Between Screener and Main Interview: It would be helpful to be able to go right from the screener to the main interview if the respondent is willing.
Screener:
- Screener Introduction: It does not work to read the screener introduction to a gatekeeper. It is more effective to save the introduction until speaking to the owner or proxy.
3. Building on Strengths: Identifying What Works and Why
Carol Emmons also led this portion of the meeting. The purpose was to identify the things that interviewers found worked well during the first pretest. The interviewers identified the following items:
- Advance letter from Alan Greenspan gave survey credibility.
- Using Federal Express for the worksheet mailing got respondents' attention.
- Asking to speak with the owner, and postponing the screener introduction until speaking with the owner or proxy, was more effective for gaining cooperation than reading the screener introduction to a gatekeeper.
- Zip code look-up table had good response time.
- Institution look-up table usually contained the branch used by the respondent and had good response time.
- Respondent incentives sometimes helped in gaining cooperation. There were differing opinions among the interviewers, but no interviewers felt that incentives harmed cooperation.
- Having definitions of questionnaire terms in the questionnaire and the QxQs made these easily accessible to interviewers and helped less knowledgeable respondents understand the questions being asked.
4. Cooperation, Refusals and Gatekeepers
Bob Bailey led this portion of the meeting. The purpose was to identify the main challenges involved in gaining the cooperation of gatekeepers and respondents with the pretest. Another purpose was to identify and share solutions to overcoming gatekeeper and respondent objections. This session covered a wide range of topics related to gaining cooperation.
Screener vs. Main Interview. In general, the interviewers found it easier to gain cooperation with the main interview than with the screener. It appears that once the owner or proxy agrees completes the screener, he or she is willing to "go the rest of the way," and complete the main interview. Exceptions to this are owners for whom English is not their first language, owners who work directly serving customers, and owners who work in noisy establishments (e.g., auto repair shops).
Interviewers felt, however, that the time between the screener and main interview should be made as short as possible to take advantage of the rapport developed during the screener. Interviewers also felt that some owners were "turned off" by the worksheets because they look complicated and time-consuming to complete.
Survey Introduction. The interviewers all agreed that the screener introduction needed to be changed. They found that simply asking to speak to the owner of the business was a much more effective way to get to speak to the owner, than first introducing themselves and the study to a gatekeeper. The interviewers recommended that we postpone the introduction until we are speaking with the person we want to interview.
There were differences of opinion among the interviewers about whether it was harmful to mention the length of the interview in the introduction. One interviewer noted that respondents sometimes held him to that amount of time. One interviewer said that a strategy he found useful was to ask the respondent for five minutes of his or her time, rather than telling the respondent how much time was needed. Another interviewer thought it better to be vague, i.e., "This will only take a few minutes."
3-attempt Rule to Reach the Owner. Interviewers sometimes found the "3-attempt rule" to reach the owner as a significant challenge during screening. In some instances, it was clear on the first call that the interviewer would never speak to the owner, yet the CATI system required three attempts to reach the owner before it allowed the interviewer to screen a proxy respondent. The interviewers feel that there needs to be a way to circumvent the 3-attempt rule when circumstances demand it.
Proxy Respondents. For some types of small businesses, such as physicians' and dentists' offices, interviewers found that there was no point in asking to speak to the owner, because the physician or dentist would never take the call. One interviewer noted that in such instances, he would say to the gatekeeper, "I want to make sure the owner knows what is going on, so please talk to the owner about this, and I will call back." Interviewers also felt that having the owner's title (i.e., Mr., Ms., Dr.) would allow them to ask for the owner in the proper way, e.g., May I speak with Dr. So-and-so.
One interviewer made the distinction between owner-designated proxies and self-designated proxies. This interviewer thought it would be helpful to have different introductions for different types of proxy respondents.
Reasons for Refusals. Interviewers mentioned the following as the most common reasons for refusing to do the screening interview:
- Respondent does not do surveys.
- Respondent stated, "Why bother?"
- Respondent wondered, "What are you really going to do with the data?"
- The introductory script in the screening questionnaire was "deadly."
- Respondent concerns about confidentiality or legitimacy of the survey.
- We were calling accountants during tax season.
Strategies for Gaining Cooperation. Interviewers mentioned the following strategies for gaining cooperation:
- Mention the project website.
- Ask for the gatekeeper's name and record it in the call notes.
- State the reason for the call as, "I'm following up on the letter you/the owner received from the Federal Reserve Board Chairman, Alan Greenspan."
- Leave a callback message with the gatekeeper if the owner was unavailable, asking that the owner call the study 800 number. (Note that some interviewers felt this was a risky strategy.)
- Use the word "confirming," i.e., "I'm just confirming basic information about firm." Try to segue right into the conversation and start asking the screening questions.
- Mention the FRB rather than the SSBF.
- Avoid using the word, "survey"; use "study."
- Start the main interview even if the owner has not yet received or completed the worksheet. Respondents who start the interview might be more likely to complete it, even if over more than one session.
- If a gatekeeper says that they received the mailing, but the owner has decided not to participate, ask why, and then try to convert.
Changes to Help Interviewers Gain Respondent Cooperation. The interviewers identified the following changes they would like to see made, in the questionnaire, call management system, and survey procedures, to help them gain respondent cooperation with the survey:
- Carry the call notes from the screener over to the main interview.
- Make the main interview available for calling more quickly after the screener is completed.
- Add the date of the respondent mailing to the preamble screen.
- Add the owner's title and or gender to the preamble screen.
- Allow more space for the type of business (SIC descriptions) on the preamble screen and the CATI questionnaire screen.
- Allow an interviewer to set an appointment for the main at the end of the screener.
Collecting Respondent Email Address. When asked how respondents reacted to being asked for their email address, interviewers commented that respondents often asked how this information would be used. Interviewers were confused about how to code a "no" response, i.e., as "no email address," or as a refusal to give the email address.
4. Unusual and Difficult Situations
Mireya Dominquez led this portion of the discussion. The purpose was to identify unusual situations that came up during pretest data collection, that interviewers felt ill-equipped to handle. The following situations were noted:
- Screener questions, "Is this the HQ or main office?" was sometimes difficult for respondents to answer. For example, an owner owns five McDonalds and we are calling him at one of them. Another example is an owner who manages real estate, and we are calling him at one property.
- The owner of the sampled business also owns other separate businesses.
- The owner's business has multiple branches.
- Wording at A.11.1 asks for street address (as opposed to city and state) but when read over the phone, having just asked for mailing address, the meaning is less clear.
- The owner is deceased.
- The business is a farm.
- The headquarters of the firm has moved since Dun & Bradstreet last updated its files.
- Reaching an owner at a peak time for that business (e.g. restaurants) is difficult. There was some discussion about setting callbacks versus letting TNMS schedule ("but TNMS doesn't know it's a restaurant"). Interviewers should ask for good time to call back, but if the gatekeeper does not offer a specific time, the interviewer should not set a general callback.
5. Main Interview Questionnaire
Bill Sherman led the discussion for this topic. The purpose was to identify questions or skip logic in the main interview questionnaire that interviewers found problematic. The interviewers made the following general comments about the main interview questionnaire.
- There is too much repetition of the business name.
- There is too much repetition of the phrase, "fiscal year ending."
- There is too much repetition of the stem, "Did you have a(n) [service] with [institution]" in sections E and F.
- Respondents who deal with more than two financial institutions gets bored with the interview.
- All the questions that ask for a comparison between two different years are awkward to read and confusing to respondents.
The interviewers also made the following comments about specific sections in the questionnaire:
| Question | Comment |
|---|---|
| B3 | If the number of owners entered at A10.1 is "2," CATI will not accept "sole proprietorship" as the firm type. |
| Question | Comment |
|---|---|
| C2 | When a husband and wife are co-owners, either may be reluctant to name a majority owner. In this situation, some interviewers found it helpful to say, "Let's just start by talking about you." |
| C30 | One interviewer thought this question (which asks if the firm is publicly traded) should be asked earlier because earlier questions sometimes make the answer obvious. |
| C32 | Change QxQ to explain that we are looking to identify the owner who has owned the firm the longest time, and to find out when that owner took ownership. |
| Question | Comment |
|---|---|
| General | Respondents rarely mentioned additional institutions after completing sections E and F. |
| F32.1 | Since vehicle loans are usually collateralized by the vehicle, the vehicle should be the first response option. |
| Question | Comment |
|---|---|
| General | Questions seem redundant with earlier questions in sections E and F. |
| Question | Comment |
|---|---|
| G11 | Respondents confuse bank with the bank holding company that provides them with credit card processing services. Interviewers should be warned about this in training. |
| Question | Comment |
|---|---|
| General | Smaller firms are unfamiliar with the term, "trade credit." Most refer to this as "having an account." Should add this and the term "invoices" to QxQ. |
| Question | Comment |
|---|---|
| General | The QxQs need to be written in shorter sentences. |
| READ 27 | Needs to be shortened. |
| M1 | Suggest re-wording as, "Did someone invest in your company?" |
| Question | Comment |
|---|---|
| P6 and P8 | One interviewer thought that using negative numbers to indicate a loss may be error-prone. |
| Question | Comment |
|---|---|
| R2 | Needs to be changed in CATI to match hardcopy version of questionnaire. |
7. Screener Issues
This section was led by Terri Kowalczyk. The purpose was to identify problems and issues with the screening questionnaire. The following issues were identified by the interviewers:
- The screener introduction needs to be revised (see also section 4 above).
- Need way to circumvent 3-attempt rule to reach owner, when circumstances dictate.
- Question A4, asking whether this is the headquarters or main location of the firm, may cause firms to screen out inappropriately.
8. Institution Look Up
This section was led by Bob Bailey. The purpose was to get interviewer reactions to the institution look-up function in the main interview CATI questionnaire. Interviewers reported that they could usually find the branch the respondent used in the look-up table. The response time for the look up was good. Finally, a surprising number of respondents knew the zip codes of their banks, but less knew the zip codes of non-depository institution sources.
9. Fading Respondents, Regaining Cooperation
Due to earlier sections taking longer than plan, this section was skipped. However, interviewers reported that most respondents were willing to complete the interview, once they started it. (See also module 4, Gaining Cooperation.)
10. Telephone Number Management System (TNMS)
This session was led by Mireya Dominquez. The purpose was to discuss possible improvements to the TNMS to facilitate the interviewers' job. Interviewers made the following suggestions:
- Change "Respondent" to "Owner" on the preamble screen.
- Add a separate field for "Proxy name" to preamble screen.
- Allow more space for SIC code label on preamble screen, and in the main questionnaire.
- Carry over the final interviewer comments collected in the screener to the main interview.
- Add a disposition for "owner refused."
- Add a disposition to request a fax be sent to the owner.
- Generally need to change "R" to "Owner" in outcome definitions.
- Add the following dispositions to SUSPEND screen:
- Owner to call 800 number
- Owner requested fax
- Answering machine
- Busy
- All initial dispositions.
- Bring the name and title of the screener respondent over to the main interview, and indicate whether the screener respondent was an owner or proxy.
- Need full set of non-contact disposition codes after starting the conversation for the screener.
- Cases assigned a "ring, no answer" disposition redelivered after too short an interval.
- Ring, No Answer Disposition: Interviewers questioned whether it is a good idea to put ring-no-answers into locating. One interviewer got a number of completes while dialing in locating. Suggest rule be reviewed.
- Add a disposition to "locating" to return case to the general interviewing location.
- When using Directory Assistance to locate people, some interviewers used "edit phone #" to change number in the TNMS. Need to decide whether to follow this protocol or put new number in call notes only.
- Suggest two locating locations: one for cases going to locating by rule and one for cases going to locating by human decision.
- Dispense with separate locations for different time zones.
11. Job Aids
This discussion was led by Mireya Dominguez. The purpose was to identify necessary improvements to the interviewer Job Aids designed for Pretest 1 as well as to identify additional job aids that interviewers thought would be helpful.
Regarding existing job aids, the interviewers suggested adding the breakpoint function to Job Aid #7: CATI Functions.
They also suggested revising the answering machine script as follows:
- Have the script emphasize the importance of the study.
- Leave the 800 number.
- Have different scripts for the screener and main interviews.
- Have a different script for missed appointments.
Interviewers also suggested developing the following two new job aids for the main study:
- Top 10 reasons to participate in this survey.
- Ways of overcoming the 10 most frequent objections to participating in this survey.
Finally, the interviewers commented that the job aids were not very accessible in the binder. Job aids need to be posted at the interviewing station.
12. Contact Materials
Pre-screening Materials. Interviewers reported that these were not very memorable to respondents. They agreed that putting the Federal Reserve Board seal on the envelope would cause respondents to pay more attention to these materials. They also agreed that the advance letters seemed to help if the respondent recalled receiving them. Finally, the interviewers liked the revised version of the project director letter that will be used for Pretest 2.
Worksheet Materials. As described above, the interviewers reported that some respondents were put off by the worksheets because they look long and complicated. Accountants seem to like the worksheets.
13. Incentives
Terri Kowalczyk led this discussion. The purpose was to collect interviewers' impressions of how helpful the incentives were in gaining respondent cooperation.
Some interviewers thought the incentives were very helpful in gaining respondent cooperation; other interviewers felt that the incentives did not make much difference. Nobody thought the incentives were harmful. A few respondents declined the incentive. The $50 appeared to be more popular with respondents than the Dun & Bradstreet Small Business Solutions package. No respondents expressed concern about their names being sold by Dun & Bradstreet. The interviewers agreed that there needs to be rules about who gets the incentive if a proxy completes all or part of the interview.
14. Interviewer Training and Training Materials
Comments and suggestions about interviewer training were made throughout the discussion of other topics. These comments are summarized in this section.
Using the TNMS:
- Setting call backs: If an interviewer calls a firm at a bad time (e.g., a restaurant at lunch time), the interviewer should try to get the gatekeeper to suggest a call back time and set a soft appointment. The interviewer should not set a general callback because the TNMS will re-deliver the case too soon.
- Sending cases to supervisor review: Interviewers needed more guidance about when it is appropriate to send a case to supervisor review.
- Writing call notes: More emphasis needed on the importance of recording the names of gatekeepers and proxy respondents in the call notes.
- Updating Firm's Telephone Number: Interviewers need clear instructions about where to record updated telephone numbers.
- SIC Code Description: Inform interviewers that "NEC" stands for "Not Elsewhere Classified."
Using CATI
- Keyboard practice: Interviewers commented that the more practice they get in training on entering responses into the CATI system, the better.
- Practice editing verbatim responses: New interviewers were unprepared for the fact that the text editor in open-ended questions works differently than in F2- interviewer comments and in the call notes ( e.g., destructive backspace, and use of "ESC" key).
- CATI Functions: More emphasis needed in training on use of the CNTL
when switching to a new respondent mid-interview, including recording the name of the new
respondent. - Question Number: Explain to interviewers how to identify the question number on the CATI screen.
Mock Interviews
- Realism: Interviewers thought that it would be helpful if the mock interview scripts used in training were more realistic and contained more of the types of challenges they encounter during actual production interviewing.
- Worksheets: Tie the worksheets to the interview more closely during the mock interviews.
Eligibility Criteria
- Interviewers were not sure how to handle cases that went out of business between the screener and main interview.
Main Interview
- Section G (Use of Other Financial Services): Need to alert interviewers to the fact that respondents often confuse their bank with the bank holding company that provides them with credit card processing services.
- Section L (Trade Credit): Need to point out the switch to a specific supplier for the questions about discounts.
- Sections R (Assets) and S (Liabilities & Equity): Need to alert interviewers to the fact that many small businesses will have a lot of zeroes in their balance sheets.
Continuous Training
- Buddy System: Assign experienced interviewers to be "buddies" to new interviewers.
- Interviewer Meetings: Provide a forum in which interviewers can share information and tips.
- Monitoring: Have new interviewers or interviewers who are having difficulty listen to experienced interviewers.