Survey Research - Part I: Questionnaires and Survey Formats

Dear research cyber-fans: for the next three chapters, we'll be taking an in-depth look at survey research. The use of surveys to collect data is one of the most popular methods of (nonexperimental) educational research. Thus, it warrants some special attention!

We'll start out with some tips for constructing questionnaire items. ***: I USE THE TERMS 'SURVEY' AND 'QUESTIONNAIRE' INTERCHANGEABLY. Next, we'll take a look at two general 'families' of popular survey item formats and the relative tradeoffs inherent in each one.

The material in this lesson is from William Wiersma's excellent text, Research Methods in Education: An Introduction, 6th ed., 1994, published by Allyn & Bacon.

Let's go!!!

Open and read the following link on Survey Research to get your feet wet.

Survey Research 1

The individual survey items should naturally link to, or emanate from, your set of sub problems, research questions and/or hypotheses.

Some survey construction experts would go a step further and argue that you shouldn't ask anything on your survey that does not specifically link to a research question or sub problem. I see the wisdom of this but have also worked with "astute, plan-ahead" doctoral students who've asked more survey questions than they initially used in their dissertations! Ah, and the "method to their madness" was this: wanting to grab this chance to build as comprehensive and thorough a database as possible, so as to "spin off" additional studies, papers, presentations, etc., after the dissertation is completed!

Thus, it is optimal if you can achieve both goals simultaneously:

1. asking all the questions you want to - "covering" your dissertation needs & perhaps beyond;
2. while at the same time, not making the questionnaire "too long," "dull to complete" or the like - so that subjects will want to cooperate and respond to all of the survey items that you have asked about.

   Here, then, is Wiersma's 'tip sheet' for survey item construction in general:

1. Try to make items as clear and unambiguous as possible. Use terminology that the subjects will understand. (Floyd Fowler, a name you'll see mentioned later on, refers to this as "establishing a common perceptual frame of reference" for the subjects. In a nutshell, this helps ensure that "they'll read into it what you wrote into it" and interpret a given word or phrase as you had intended - thereby increasing reliability of responses. Example: "How many members of your household do you have?" Some might count just immediate family - i.e., parents, children, spouses. Others might count more distant relatives if they are also living in that particular home. Another example: "Which magazines do you subscribe to?" Does one count, say, TV Guide? One doesn't 'read' it in the same way as other magazines; e.g., cover to cover. Yet it is a periodical that can be purchased by subsription!). Avoid vague words, technical terms, and jargon to the greatest extent possible.
2. Include only a single concept per item. (*** Qualitative friends! We have already visited this issue in our discussion of writing "good" interview questions! We suggested that you avoid "double-barreled" items, or those which sound like one question but really contain two (or more) embedded questions!) Example: "Are you in favor of minimum competency testing and career ladder teacher incentive plans for newly hired teachers?" How does one interpret, say, a "yes?" That the respondent agrees with a) the testing; b) career ladders; or c) both?!
3. Try to avoid the use of "leading" questions. These are questions with implied assumptions or anticipated outcomes. Often, such items indicate a 'preferred' response ... however subtly! Example: "Are you in favor of relaxed discipline in the schools, even though such discipline may undermine the moral development of youth?"
4. Try to avoid questions that are "loaded" with social or professional desirability. In essence, do not ask questions so that certain responses would lead the respondents to 'disapprove of themselves.' An example would be to ask a teacher: "Do you have difficulty maintaining a positive learning environment in your classroom?"
5. Try to avoid questions that demand 'personal' or 'delicate' information - unless it is essential to the particular variables of your sub problems, research questions, and/or hypotheses. Such items may range from questions on exact age and income, to questions asking the respondent's extent of involvement in illegal/immoral activities (i.e., drug use).
6. Only request information that the subject is actually able to provide. All items should "fit" the informational background of the respondents. (Otherwise, the reliability of your study may suffer due to misinterpretation, "guessing," skipping those items, etc.)
7. Try to make the reading/comprehension level of the items appropriate for the subjects. To the greatest extent possible, use "soft" words as opposed to "hard" words. For example, when surveying teachers you might opt for the 'more neutral' and 'less emotionally loaded' terms of 'correction' or 'corrective action,' as opposed to 'punishment' of students.
8. All other things being equal:
   1. Shorter items are generally preferred to longer items; &
   2. Simpler items are generally preferred to more complex items. This would imply that, rather than opting for a single, detailed, complex survey item, your best bet would be to try & break it up into two (or more) shorter, more focused survey items.

   
   

9. When requesting quantitative information, ask for a specific number (i.e., an actual frequency), rather than an average. For example, ask: "How often during the past month did you help your child with his/her homework?" rather than, "On the average, how many times a month do you help your child with his/her homework?"
10. To the greatest extent possible:
    1. Avoid negatively worded items; &
    2. Do not use "double negatives." Example of "a:" "Which of the following instructional strategies do you not use?"

     

- - -

Before leaving this topic of general survey guidelines, it is perhaps even more important to consider the overall issue of whether or not to construct one's own survey in the first place!

1. Find and stick with an existing instrument? vs.
2. Construct your own.

Proponents of #1 argue efficiency: why reinvent the wheel if what you need is already out there!

And, this makes eminent sense IF what's out there also ideally fits your individual needs (populations, key variables, research questions, and so forth)!

If that is not the case, and/or if you cannot get the author's permission to use or adapt his/her/their instrument to your unique needs - > PLRSDR don't hesitate to design your OWN survey!

There's a subtle 'bias' that we often carry with regard to self-constructed instrumentation. I sometimes refer to it as the "slick marketing packaging bias." That is:

"Oh, gee, unless it's been 'done up and put together' by a big educational research and testing firm, and piloted on 1,000's of cases across the country, why, it won't be as rigorous!"

Not so! Quite the contrary: by developing, pilot-testing and revising your own survey, as custom-tailored to your own unique needs, research questions, subjects, settings - you are actually thereby increasing its credibility and relevance - i.e., VALIDITY!

We'll be talking a bit more about pilot testing next time. Until then: keep an open mind and trust your abilities to 'start from scratch' and devise a survey that's right for you!

II: General Types of Survey Formats

There are two basic broad "families" or types of these:

1. Selected-response/forced choice: where the subject chooses from two or more options that are provide

   Advantages:

   1. Enhances consistency of responses (since those choices are spelled out, "already there" and sometimes even pre-defined (e.g., what exactly is meant by "rural school setting");

   2. Easier to tabulate and code these data into a database - generally, code numbers can be pre-assigned to each choice, thus making the database readily quantifiable and paving the way for trained data-entry personnel to enter the survey responses into the database;

   3. Somewhat easier and quicker for subjects to respond to (as opposed to having to 'think through' and 'fill in, in their own words'). Of course, as we've learned in Qualitative, 'ease of response' may not necessarily be the best or only objective. We may need and want the 'richness and depth of detail' that would necessitate such 'think time and effort' on the part of our subjects to be able to provide us with a response in their own words. Still, ease and efficiency are two criteria that cannot be overlooked in any survey design or construction issue. Whether or not they are the primary criteria depend on the specific needs and circumstances of the study. If they are key, then the fixed-choice responses may be one way to attain these criteria.
   
   

   Disadvantages:

   1. Due to the 'pre-existence' of the allowable choices, they can sometimes 'box in' or constrain the respondents regarding the breadth/richness/depth of the response (this can be partially overcome by making sure there is as complete and relevant a list of choices as possible - pilot testing can help here);

   2. Due to (a) above, construction of "good" fixed-choice items may be relatively more time-consuming (including perhaps even more than one pilot to ensure completeness, clarity, etc.) than simply posing the same general question in open-ended format.

2. Open-ended items: for which the respondent furnishes the response in his/her own words, being provided a blank space on the survey to do so.
   
   
   Advantage:
   1. Such open-ended items allow relatively more freedom of response, since respondents are free and welcome to provide attitudes, perceptions, beliefs, etc., in their own words. (*** Qualitative friends! You will recognize this as a golden opportunity for "grounded theory," or allowing the key themes, variables, concepts, etc. to emerge directly from the subjects themselves rather than being 'pre-imposed' by the researcher.)
   Disadvantages:
   1. Both questions and responses are sometimes subject to misinterpretation. Different subjects may 'read different meanings' into a question; and the researcher may similarly misperceive, misclassify, etc., the written open-ended responses. Such misinterpretation may greatly hamper both the validity and reliability of the study.

   2. Also, given the opportunity to 'respond in your own words,' some subjects may include irrelevant information in a given response.

   3. This in turn increases the relative length of time needed to sort, code, and classify such open-ended responses, as compared with tabulating fixed-choice responses. As we have been learning in our Qualitative cyber-adventure, there may be a great deal of subjectivity inherent in creating and validating a coding or classification scheme for one's qualitative data. This is not necessarily a negative in and of itself; furthermore, established procedures exist for increasing the 'credibility' of one's coding scheme: i.e., extensively piloting the coding procedure with expert judges beforehand, as well as using multiple raters and running reliability (extent of coding agreement) checks. Still, such procedures inherently increase the length of time, energy and commitment of resources to the study.

1. For example, the following question would be seen as unstructured:

"What do you think of the honors program in mathematics?"

- > Please note the potential for grounded theory in the above, qualitative fans! The researcher is allowing for either "kudos" or "gripes" to "emerge," by not cuing for directionality. In addition, he/she (researcher) is also not cuing for any particular aspect of the program. Rather, the researcher is 'setting the stage' for the subject to choose: 1) what part(s) or aspect(s) of the math honors program he/she wants to mention; and 2) in what sense (as a positive, negative, suggestion for improvement, etc.).

 

2. Now, let's look at the same idea in partially structured form:

"What do you think about the effectiveness of the mathematics honors program relative to advanced placement in college?"

Here, the researcher has 'prepicked' a particular aspect of the honors program -- i.e., its relationship to future college placement -- and is asking the subjects to evaluate its effectiveness. Again, 'anything goes' with regard to directionality -- they are free to praise or 'bash' the program!

-- but the "boundary" or "fence" has been set regarding which part of the program they are to be thinking about and evaluating.

 

3. Finally, here is a structured version of a related survey item - please note the predetermined, selected-response format:

Do you feel that the present honors program in mathematics should be:

1. continued without modifications?
2. continued but modified?
3. discontinued?

Such a structured, selected-response item would presumably be accompanied and/or preceded by a clear operational definition of that particular mathematics honors program. This would be done to, again, help ensure what Floyd Fowler has termed a "common perceptual frame of reference" for the respondents. They all understand exactly which program is being referred to and that they are to be thinking of as they answer the related questions.

In addition, for the totally structured format of the preceding item, the directions would specify that the subject is to choose ONLY the SINGLE response for each item which MOST CLOSELY represents his/her opinion. There would be no latitude for any written-in responses. (For partial structure, the subject might be asked to write in, say, a brief rationale for his/her choice.)

- - -

Next time out, we'll continue our "survey odyssey" with a bit more in-depth look at some subtypes of fixed-choice items. The most popular of these is the Likert-scaled item. We'll also talk about the nature and role of pilot-test procedures in establishing the validity and reliability of questionnaire items. Along the way, we'll look at some tips for constructing 'good' cover letters in the case of mailed surveys. Finally, we'll 'carefully confront' the hope (bane?!) of every survey researcher: issues involving the response rate and problems of nonresponse bias.

- - -

Go on to Assignment 1
or
Go back to Survey Research Part I: Inquiring Minds Want to Know: Questionnaires and Survey Formats

E-mail M. Dereshiwsky at statcatmd@aol.com
Call M. Dereshiwsky at (520) 523-1892

Copyright © 1999 Northern Arizona University
ALL RIGHTS RESERVED