Is It Wrong? Handling Missing Responses in IRT

Abstract

Estimation of examinee ability under Item Response Theory is affected by how omitted test items are handled. This paper examines omission patterns in an operational third-grade math assessment and investigates the effects of different treatments of missing test data on ability estimation through simulations. Surprisingly, there was little correlation between omission rate and item difficulty; and while lower-ability students did omit short answer and extended response items more frequently than high-ability students, high-ability students actually skipped multiple choice items more frequently than low-ability. According to simulations, treating omitted items as incorrect, while pervasive, significantly biases ability estimates downward, particularly for high-ability examinees. Treating omitted items as if they had not been administered has the smallest effect on ability estimation, but could encourage examinees to game the system on high-stakes tests. Treating missing responses as half-correct also performs fairly well, but suffers from the same weakness to devious examinees. In the end, treating omitted responses as wrong for operational tests may be the best option at the moment until further studies are available on when and why students skip test items.