Modeling item bias in fixed-item tests and computerized adaptive tests

Chen, Dandan

Permalink

https://hdl.handle.net/2142/120189 Copy

Description

Title

Modeling item bias in fixed-item tests and computerized adaptive tests

Author(s)

Chen, Dandan

Issue Date

2023-04-18

Director of Research (if dissertation) or Advisor (if thesis)

Zhang, Jinming

Doctoral Committee Chair(s)

Zhang, Jinming

Committee Member(s)

• Anderson, Carolyn
• Kern, Justin
• Xia, Yan
• Shin, David

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• differential item functioning
• DIF
• fixed-item tests
• computerized adaptive tests
• CAT

Abstract

Education equity and fairness of assessment are two of the most important topics in education. Bias is a related concept, where there is construct bias, method bias, and item bias (e.g., van de Vijver & Poortinga, 1997; Werner & Campbell, 1970). Measurement invariance is another one, essential in modern standardized assessments based on item response theory (IRT; Richardson, 1936). It includes (1) configural invariance, (2) weak invariance (or metric invariance), (3) strong invariance (or scalar invariance), and (4) strict invariance (Wu et al., 2007). It ensures the comparability of scores across different examinee groups. An operational definition of the scalar invariance is precisely the definition of differential item functioning (DIF; Berk, 1982), a framework that underpins modern methods to investigate bias that possibly exists in assessment items. Despite its long-standing literature and wide application in the testing industry, DIF remains an important area for psychometric research and the discussion concerning DIF detection methods keeps evolving. This dissertation aimed to highlight the limitations of a few commonly used DIF methods in psychometric practice and suggest alternative approaches to improve DIF detection accuracy. Chapter 1 provides an extensive review of the DIF methods having been widely applied in either the testing industry or academic literature. The methods include those following the single-level or the multilevel framework. Chapter 2 reveals key patterns associated with the cutoff value of the RMSD approach, which is a popular DIF method for analyzing cross-country score comparability in international large-scale assessments. Based on Type-I error rates from simulation, a polynomial regression was proposed to predict the appropriate cutoff given the number of groups to be analyzed and significance level. Chapter 3 compares the performance of the RMSD approach with four other approaches designed for multi-group DIF analysis. When compared with a few acceptable methods in the evolving multi-group DIF literature, the RMSD approach was demonstrated to be optimal when used along with the predicted cutoff. Chapter 4 concerns a multilevel framework for DIF detection with data from computerized adaptive tests (CAT), where DIF remains understudied. It conceptualized how between-item dependency could be reframed into between-examinee dependency, and proposes a multilevel model to obtain more accurate DIF results after accounting for this dependency. Chapter 5 lays out a few directions for future research, including developing a global effect size measure across DIF methods, a general DIF approach based on a versatile framework named structural equation modeling, and a shift of the statistical framework for DIF detection.

Graduation Semester

2023-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Dandan Chen

Owning Collections