Using differential item functioning analysis to determine differential interpretations of survey questions

Abstract

The application of Differential Item Functioning (DIF) analysis to noncognitive areas such as psychological and attitudinal measurement is a natural extension of the current research and is the focus of this study. This study uses DIF analyses to examine differential item functioning of Likert response items formatted from survey instruments.

Data for this study were drawn from the responses of 22,651 parents of eighth grade students on 26 questions from the National Educational Longitudinal Survey of 1988 (National Center for Education Statistics, 1990). A differential item functioning analysis of these 26 questions was conducted to determine if different groups of parents interpreted the survey questions differently. To detect DIF in this survey, logit models were utilized. Total score for each substantive area was employed as the conditioning variable.

The study revealed that DIF analyses can be conducted with noncognitive instruments using multiple factors simultaneously to yield important information about the questions. In this survey, several of the questions were functioning differently for different groups of parents based on their race or whether their child attended a city school. For a few questions, the interaction term between race and urbanicity was significant in the model thus providing more insight into understanding DIF.

The focus of this study was to exemplify three points: (a) that instruments other than cognitive tests can exhibit DIF, (b) that interaction terms are important to include when testing models, and (c) that studying groups defined on characteristics not commonly used in current DIF studies can provide better insight into DIF. Survey instruments are a commonly used tool in evaluations and other studies to gather information regarding opinions and attitudes from large groups of people. Because the responses from surveys are often reported in an aggregate form, it is often the case that the survey questions are being summarized uniformly for the entire sample. An assumption is made in most cases that the survey questions are being interpreted consistently across the population of respondents. One method of verifying this assumption is to utilize differential item functioning analysis. This study demonstrates the utility of DIF for this purpose.