Survey content and structure
Novice researchers often assume that developing a questionnaire is an intuitive process attainable by virtually anyone, regardless of their level of research training. While it is true that questionnaires are relatively simple to construct, developing an instrument that is valid and reliable is not intuitive. An instrument is valid if it actually measures what we think it is measuring, and it is reliable if it measures the phenomenon consistently in repeated applications.24 By following a few basic guidelines, those with limited research training can develop survey instruments capable of producing valid and reliable information. The 3 primary concerns for developing appropriate questions (items) are: (1) response format; (2) content; and (3) wording and placement (see Figure B, Survey Questions; and Figure C, Designing and Formatting the Survey).
Format
Questionnaires generally use a closed-ended format rather than an open-ended format. Closed formats spell out response options instead of asking study subjects to respond in their own words. Although there are many reasons for using closed formats, their primary advantages over open formats is that they are more specific and provide the same frame of reference to all respondents, and they allow quantitative analysis. A disadvantage is that they limit the possible range of responses envisioned by the investigators. Questionnaires with closed formats are therefore not as helpful as qualitative methods in the early, exploratory phases of a research project.
Closed-ended items can be formatted into several different categories (classes) of measurement, based on the relationship of the response categories to one another. Nominal measurements are responses that are sorted into unordered categories, such as demographic variables (ie, sex, ethnicity). Ordinal measurements are similar to nominal, except that there is a definite order to the categories. For example, ordinal items may ask respondents to rank their preferences among a list of options from the least desirable to the most desirable.
Survey items that ask for respondents(delete apostrophe) to rank order preferences are often a more useful than items that state, “check all that apply.” While checking all relevant responses may be necessary for certain items, such questions often lose valuable information as they can only supply raw percentages without supplying any comparison between responses. If a survey uses a rank order response, it enables determining the relative importance of the different categories during data analysis Table 1.
Two additional tools used on questionnaires are continuous variables and scales. Continuous variables can be simple counts (eg, the number of times something occurred) or physical attributes (eg, age or weight). A general rule when collecting information on continuous variables is to avoid obtaining the information in ranges of categories unless absolutely necessary. Response categories that reflect ranges of responses can always be constructed after the information is gathered, but if the information is gathered in ranges from the start, it cannot later be expanded to reflect specific values.
Scales are used by survey researchers to assess the intensity of respondents’ attitudes about a specific issue or issues. Likert scales are probably the best known and most widely used for measuring attitudes. These scales typically present respondents with a statement and ask them to indicate whether they “strongly agree,” “agree,” “neither agree nor disagree,” “disagree,” or “strongly disagree.” The wording of the response categories can be changed to reflect other concepts (eg, approval or disapproval), and the standard 5-response format can be expanded or abbreviated if necessary.
There are no hard and fast rules for determining the number of response categories to use for scaled items, or whether to use a neutral category or one that reflects uncertainty. Research indicates that the reliability of respondents’ ratings declines when using more than 9 rating scale points.25 However, the reliability of a scale increases when the number of rating scale points is increased, with maximum benefit achieved with 5 or 7 scale points.25,26 Since the objective of using scales is to gauge respondent’s preferences, it is sometimes argued that a middle point or category of uncertainty category should not be used. Odd-numbered rating scales, however, conform better with the underlying tenets of many statistical tests, suggesting the need for including this category.29 As the number of rating scale points increases, respondents’ use of the midpoint category decreases substantially. 30 Thus, based on the available literature, it is generally advisable to use between 5 and 7 response categories and an uncertainty category, unless there is a compelling reason to force respondents to choose between 2 competing perspectives or alternatives.
Content
Items should not be included on questionnaires when the only justification for inclusion is that the investigator feels the information “would be really interesting to know.” Rather, for each item, you should ask yourself how it addresses the study’s research question and how it will be used in the data analysis stage of the study. Researchers should develop a data analysis plan in advance of administering a questionnaire to determine exactly how each question will be used in the analysis. When the relationship between a particular item and the study’s research question is unclear, or it is not known how an item will be used in the analysis, the item should be removed from the questionnaire.