To close out our survey research discussion, dear cyber-partners, we'll
take a look at the following topics:
While most attention is naturally focused on the "proper development"
of the survey instrument itself, researchers need to also keep in mind
the "power of first impressions" conveyed by the cover letter!
B. Example of a Survey Cover Letter
The following example is an actual survey cover letter composed by William
Wiersma in his research commissioned by the Appalachia Educational Laboratory
(AEL).Wiersma served as the external evaluator of the related study. As
you review this letter, please compare it to the preceding list of qualities
of 'good' cover letters (Part "A," pgs. 2-4).
October 5, 1998
Dear [name]:
The Western Michigan University Evaluation Center has been engaged
by the Appalachia Educational Laboratory (AEL) to be its external evaluator.
As part of our evaluation, we are surveying people who are receiving
AEL materials and services. There are two sections to the questionnaire;
the first is a section that deals with R & D Notes, whose masthead
is reproduced at the top of Section 1. You have been identified as a
recipient of R & D Notes and familiar with it. The second section
of the questionnaire deals with your perceptions of AEL materials and
services in general.
The information provided through the questionnaires will be presented
to the AEL Board, the federal government, and AEL staff to help them
improve AEL materials and services. Your responses to this study will
be confidential; no individual will be identified with his or her
responses. The number on the questionnaire is a code so that we
can identify those who have responded. This is to reduce the cost of
follow-up and to eliminate the disruption of follow-up for those who
have returned the questionnaire.
Your response is very important to the success of this evaluation.
The information you provide is important to AEL, not only for program
planning, but also in dealing with the funding agency. Completing the
questionnaire should require no more than 20 minutes. We very much appreciate
your completing and returning the questionnaire by October 26,1992,
in the enclosed, postage-paid envelope.
Sincerely yours,
[signature]
William Wiersma, Ph.D.
External Evaluator for AEL
(typist's initials)
enclosure
More examples: Look at these examples and consider whether they are
good examples of survey cover letters.
Indianapolis
Parochial School Survey
Cumberland
County School to Career Employment Survey
II. Issues Regarding Non-Response
Of course, we hope this doesn't happen to us! That all of our
selected survey subjects will cheerfully comply and return a completed
survey within our specified time period!
Ah, but now for the reality of the "respondent paper
chase!"
Let's hope that 'tracking 'em down' isn't quite this arduous...!
A. Why Non-Response Is a Problem
The greater your non-response total or percentage, dear
cyber-researchers, the bigger the danger of the following:
That your survey non-respondents are systematically
different in some way from the respondents regarding the target
outcome, key variables of interest in your study.
It all comes down to this key issue!!! In fact, this is
so important that I recommend that all dissertation writers planning
to use survey research as all or part of their study include the preceding
point as one of their Chapter One Limitations (threats to validity)!
Please note the two critical points in this issue:
- Systematically different from respondents?
If so,
if the non-respondents 'aren't like' the respondents, then we cannot
safely assume that they 'represent the SAME population!' This
is a clear and present danger even if we 'played by the rules' and
drew a probabilistic (i.e., simple random, systematic, stratified,
or cluster) sample. If they end up being 'that different,' then for
all intents and purposes they represent a 'different sub-group' than
the respondents!
AND....
- Does the 'difference' occur with regard to the 'target outcome
variable(s) of interest' in the study?
If those same non-respondents
happen to support the Democratic gubernatorial candidate, while the
respondents were by and large in support of the Republican candidate,
that would be a problem for your study if your topic deals with political
voting preferences. On the other hand, if your topic concerns perceptions
regarding favorite books or magazines, the above difference might
be of little or no concern (except, of course, if it can be shown
that political party affiliation and magazine/book preference are
themselves correlated!)! I think you see the key point here. Did the
non-respondents deliberately choose to withhold the key information
about a 'sensitive issue,' such as say, abortion or capital punishment,
because in fact they differ radically in attitude from those who did
choose to reveal their attitudes on the survey? Then we would have
a problem, due to a systematic bias in the survey responses in our
database. They simply would 'miss' or not capture a key 'chunk' of
attitude/willingness to share response that actually exists out there
in our total target sample (and not just those who cooperated with
us and responded).
B. Minimum Return Rate and "Post Hoc" Remedial Steps
What, then, is a 'good bare-bones' minimum return rate
to help minimize the danger inherent in the preceding two key sub-issues?
Again, I can at best give you a 'general all-purpose rule
of thumb' that has been established and enforced by universities, professional
journals and conferences, etc. But I will also caution you to do a bit
of homework first, because this rate varies widely by specific academic
discipline! Check prior studies in journals in your field to see
if, and what, the 'topic-specific' minimum-acceptable survey return
rate is!
The general one seems to be: 70%
Barring any discipline-specific different benchmarks;
i.e., that you can 'convince us,' as your thesis or dissertation committee,
that 'it's different in your field,' this is the minimum survey return
rate we'll 'hold you to' in analyzing your thesis/dissertation data
at CEE! (The same benchmark was in effect at the University of Massachusetts
- Amherst, where I got my Ph.D.)
Again, remember! This rate, or a content-area specific
different rate, is "bare minimum!" Even in the above, 30% is
a hefty 'chunk' of your desired target sample basically 'unaccounted
for!' That percentage certainly leaves the door open to the possibilities
that "they could be different regarding the study variables!" So - while
you can 'breathe a sigh of relief' that you got 'enough' surveys back
so that you can proceed with analyzing your thesis/dissertation data,
keep in mind that "more is ALWAYS preferred to less!" In reducing
that potential 'window of systematic bias' or difference between your
respondents and non-respondents!
So - don't rest on your laurels just yet! Your overriding
goal should be to maximize your return rate - period! As close to 100%
as you can get it! We'll look at some tips for maximizing this return
rate in the following subsection.
For now, let's 'worst-case scenario it' and get that extreme
out of the way! What if you've followed all the sage advice to come
in the following subsection, did everything within your power to get
that survey return rate up to the 70% overall or discipline-specific
benchmark .. and it's just no go...???
It will then be up to you to 'convince' yourself and your
thesis/dissertation committee that you have some evidence showing that
the non-respondents, while a 'hefty percentage' in number, are in fact
not systematically different from the respondents regarding those key
target variables of interest. In other words, you'd be arguing that
sure, your absolute number of respondents is 'low,' but those who didn't
respond to your survey would have been quite similar on the key issues
covered therein.
How do you do this? Not easy, I'll admit! But there are
at least 2 possible avenues to consider to 'make your case' if you simply
must go this route:
- Gathering evidence from existing archival databases.
This one may be a 'long shot,' but a disseration candidate of mine
used it to positive advantage! If someone else has a database, i.e.,
census data or prior survey data on a population from which your sample
subjects were also drawn, and you can locate statistics to show a
similarity on, say, background demographic or other variables, then
you might be home free.
- Following up with a small sample of non-respondents.
This one is generally more promising but will also probably require
every ounce of your perseverance! It requires simply 'gritting your
teeth and going after' those who failed to respond to your survey
and either getting them to eventually complete it or perhaps (as in
another dissertation committee on which I chaired) getting them to
'verbally walk through' the same items or issues as you asked about
on the paper-and-pencil version of the instrument. In either case,
you then take the two subgroups: 1) the original respondents and 2)
the 'eventual reluctant (non) respondents,' and compare the two sets
of answers. Again, your goal or hope is to show that there is no 'significant
difference,' be it assessed quantitatively (i.e., an independent-samples
t-test or Mann-Whitney U test) or qualitatively (by comparing the
types and frequencies of open-ended written/spoken responses) on key
variables between the two groups. If you can indeed demonstrate this,
you are home free in that you have some evidence that the non-respondents,
while admittedly a larger number/percentage than you'd have liked,
'are not systematically different from the respondents regarding the
key variables of interest.'
C. Tips for Increasing Survey Response Rates
Again, may you not find yourself in the preceding precarious situation
to begin with...! With good 'ex ante' planning and strategizing, there
are some things you can do to help ensure a hefty response rate!
1.
- "Before the Fact" Issues
Some of these have already been alluded to in the prior two survey
lesson packets, as well as the tips on composing a 'good' cover
letter on pgs. 2-4. In other words, careful thought and attention
to quality and appearance of the 'whole package' will play a role
in the target survey respondent/recipient's propensity to 'go along'
with your request to complete and return it!
Wiersma highlights three of these in particular:
- Any ensuing 'halo effect' to you of 'being positively
regarded by the survey recipient.' This does not necessarily
mean that he/she knows you personally. This "personal impression"
issue can mean anything from how well you've stated the purpose
of your survey research, to the identity of your 'sponsoring'
organization and/or thesis/dissertation chair, to your own title/affiliation,
to the care and attention you've paid to grammar, punctuation,
and sentence structure in your writing - to, most likely, an overall
'holistic' judgment subsuming all of the preceding factors!
- Your indication to the respondent of his/her role and
importance to the survey research. This too is a holistic
type of "judgmental variable." Whether consciously or subconsciously,
your survey subject will be making a decision as to his/her "buy-in"
into the survey. Have you captured his/her attention and interest?
Have you convinced him/her of the value of the responses to be
offered in a sincere, 'non-patronizing' manner? and so on and
so forth!
- Your expression of appreciation to the respondent.
Ah yes, the 'good manners' issue again! Evaluated, at least on
some level, by the survey recipient as to sincerity!
Again, the preceding are to be considered along with the general
'quality control' issue of good survey construction and design that
we've discussed at greater length in the two preceding packets!
Also, you need to know that "jillions" of individual controlled
studies have been done which have 'tinkered' with isolated factors
or parts of the survey package itself: even such things as the color
of the paper used for the cover letter, and the type and size of
print. It may not surprise you to learn that, "bottom-line," none
of these individual factors in and of itself had as much 'significance'
as the overall attractiveness, ease of reading and appearance
of the whole package. Trite as it may sound, then, you should
aim these more general, "holistic" qualities, including running
them by a pilot sample (more on that in a bit!), rather than
worrying too much about the individual factors. Guess it confirms
what we already knew: people make an overall value judgment, as
opposed to getting too lost in the 'pieces!'
One procedural factor has
been found to make a significant difference, if it is feasible:
contacting the respondent prior to mailing him/her the survey. It
hasn't seemed to matter too much whether this 'pre' contact is done
by letter, postcard, or telephone call. Perhaps it helps to increase
the 'respondent's buy-in and perceived importance of his/her responses
to the overall goals of the survey,' as per "ex ante factor b,"
in our preceding discussion.
But what if you find that you still don't have the requisite minimum
number/percentage of returns as of your specified target date? Don't
despair! There are still some things you can do to follow up 'after
the fact' to try and edge that return rate upwards!
- "After the Fact" Issues
- Re-send a reminder letter 7-10 business days after the deadline
for return has passed. If you have used a numeric coding scheme
to label and track each survey, such as was mentioned by Wiersma in
his cover letter above, you will be able to readily identify the non-respondents
while at the same time preserving the confidentiality of the respondents.
- Include an extra copy of the survey, along with a self-addressed,
postage-paid envelope. This is done as a 'convenience,'
in the event that the subject has perhaps misplaced the original and
sincerely desires to respond.
You might also consider telephone
calls to the non-respondents, although this can be rather time-consuming
and expensive in the case of long-distance dialing. Also, repeated follow-ups
(3rd, 4th and other 'waves') are occasionally necessary, although it
has repeatedly been shown that they result in successively smaller percentages
of returns - and granted, you may have little choice if you are still
looking to 'hit' that minimum return rate!
Finally, keep in mind that while you are partially 'plugging' the
original 'threat to validity' or 'limitation' of non-response bias
(the key issues raised earlier in our discussion regarding non-respondents),
with such lapses of time and successive waves of return you may be
subtly introducing a slightly different 'threat to validity:' "Are 'late
responders' systematically different from 'on-time responders' with
regard to the key variables of interest?' The more time that
passes between the original survey return date and the late returns,
the greater the danger from this threat. As before, it
will be incumbent upon you to marshal evidence of lack of difference
between the two subgroups of respondents (on-time vs. late) in the same
way as for the respondents vs. non-respondents (i.e., the 2 possible
plans of action identified earlier).
Bottom line time: Nothing you didn't already know! It's far
better to do everything within your power to ensure a hefty return
rate "up front and on time," with your own careful planning and attention
to such factors as quality and appearance of the survey and supporting
materials, than it is to hope to 'save yourself' with non-respondent
follow-up after the fact!
Tips on Minimizing Non-Response
(Mailing)
And ... by way of transitioning to our final topic of discussion today
regarding surveys, a key part of reaching this goal is, in effect, a
'dress rehearsal.' You might plan for a PILOT TEST with
a 'holdout' sample of subjects ("similar to," i.e., perhaps a random
draw, from the eventual study population, except that these subjects
will not subsequently become part of the survey recipient sample --
"blast from the past," Intro and Res. Des. friends -- sound like "sampling
without replacement?" from Population and Sampling?!) and/or prior review
by a balanced selection of a panel of expert judges to provide input!
Such "dry runs" and/or "ex ante review by experts" can go a long way
in helping you self-assess how well you attained those target holistic qualities
of quality survey construction, ease of reading of materials, attractiveness
of appearance of the whole package, etc.!
