Q-matrix optimization for cognitive diagnostic assessment

Chen, Cong

Permalink

https://hdl.handle.net/2142/98150 Copy

Description

Title

Q-matrix optimization for cognitive diagnostic assessment

Author(s)

Chen, Cong

Issue Date

2017-05-03

Director of Research (if dissertation) or Advisor (if thesis)

Zhang, Jinming

Doctoral Committee Chair(s)

Zhang, Jinming

Committee Member(s)

• Anderson, Carolyn
• Chang, Hua-Hua
• Culpepper, Steven

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Q-matrix
• Cognitive diagnostic assessment

Abstract

Cognitive diagnostic assessment is a growing area in psychological and educational measurement with the advantage of providing diagnostic profiles (mastery/non-mastery of measured attributes) for examinees, giving insights for classroom teaching and student learning. Central to the successful implementation of a cognitive diagnostic assessment is the Q-matrix, the structure that specifies item-attribute relationships. However, the Q-matrix is prone to misspecification, given that it is often constructed based solely on human opinions. This thesis uses three research studies to investigate key issues of Q-matrix optimization for cognitive diagnostic assessments. The first study investigates the effects of Q-matrix misspecification on the classification accuracy and consistency of diagnostic results. The two types of Q-matrix misspecifications examined are Q-entry misspecification (which includes three levels of misspecification: 10%, 20% and 30%), and attribute misspecification (which includes attribute exclusion and attribute inclusion). The results of a simulation study show that both Q-entry and attribute misspecification significantly deteriorate the accuracy of classification and the consistency of diagnostic results. In addition, the two classification accuracy and consistency indices have the potential to be useful in identifying possible attribute misspecification (e.g., attribute inclusion) of Q-matrix in empirical analyses. The second study provides a systematic performance evaluation of the three most commonly used Q-matrix validation methods: the sequential EM based δ-method (de la Torre, 2008), the Bayesian estimation method (DeCarlo, 2012), and the nonparametric Q-matrix refinement method (Chiu, 2013), with both basic and complex assessment design factors. The results of two simulation studies reveal that the Bayesian estimation method outperforms the other two methods in terms of recovering the misspecified Q-entries across various conditions. The performance of the three Q-matrix validation methods is also affected to different degrees by various assessment design factors, among which the data generation model is the most critical. The third study proposes a two-stage cross-validation method that combines the strengths of the nonparametric refinement method and Bayesian estimation techniques for improving Q-matrix validation accuracy and computation efficiency. The results show that the proposed method can effectively optimize Q-matrices that are possibly misspecified in both simulation and empirical data settings.

Graduation Semester

2017-08

Type of Resource

text

Permalink

http://hdl.handle.net/2142/98150

Copyright and License Information

Copyright 2017 Cong Chen

Owning Collections