Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type

Yi, Yeonsook

Permalink

https://hdl.handle.net/2142/42124 Copy

Description

Title

Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type

Author(s)

Yi, Yeonsook

Issue Date

2013-02-03T19:16:27Z

Director of Research (if dissertation) or Advisor (if thesis)

Davidson, Frederick G.

Doctoral Committee Chair(s)

Davidson, Frederick G.

Committee Member(s)

• Zhang, Jinming
• Bowles, Melissa A.
• Templin, Jonathan
• Tremblay, Annie

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• cognitive diagnostic assessment
• cognitive diagnostic model comparison
• general modeling framework
• language test data
• Test of English as a Foreign Language (TOEFL) Reading and Listening
• university French placement test
• elicited imitation task
• resampling
• polytomous
• networking in language testing
• Examination for the Certificate of Proficiency in English (ECPE) Grammar

Abstract

This dissertation is based on two major projects, cognitive diagnostic model comparison and implementing cognitive diagnostic assessment in an institutional test. In the first project, five cognitive diagnostic models are empirically compared for language test data under a unified general modeling framework. The models are applied to three sets of data, TOEFL Reading, TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data and functioning. The item-association root mean squared error values and multiple information criteria all indicate that the general model (LCDM) and the compensatory-RUM model are the best fit to all three test data sets used. The functioning of the models examined through multiple indices also unanimously confirms these fit indices. Based on these results, a discussion follows to argue that the general modeling framework is optimal for language assessment data due to its much greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM (Fusion) models are also compared and the differences are analyzed. Cognitive diagnostic assessment (CDA) has gained attention in language testing since late 90’s. A few models of this new assessment method have been applied to the response data of different language tests, yielding encouraging results in general. Most of these empirical studies used large-scale, standardized tests and retrofitted to these existing tests. The second research study in this proposal was an effort to go beyond this limited research context of previous studies of CDA in language testing. Using a new psychometric cognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicited imitation task) as well as constructing the EIT test with a CDA implementation in mind from the outset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment in an institutional placement test. In doing so, the study employed a statistical method (resampling technique) as a way to resolve the issue caused by a small size of a test-taker pool at an institution, which is usually smaller than the optimal size for CDA implementation. It also tried to analyze polytomously scored response data. The study achieved a success at estimating with polytomous response data that were scored with a three-point scale (i.e., zero to two points). Though it was a limited success (in that more complex rating scale could not be analyzed), it was the first success in estimating with polytomous response data in the context of CDA research in language testing. The analysis results of the study also provide many insights and implications for the process of Q-matrix construction, grain size of attributes, appropriate task types and item types for cognitive diagnostic assessment as well as appropriate cognitive psychometric models for differing contexts of CDA implementation. The study also rediscovers or introduces the usefulness of bootstrap resampling method as an approach that is gaining popularity even in areas where only traditional quantitative methods are usually employed. Also, the networking occurred in this project between students in different specializations could be established as a new networking model in language testing. Considering such a collaboration is very much needed for implementing a relatively new measurement method in a specific knowledge domain, the co-work attempted in this project could serve as a model for implementing CDA in language testing.

Graduation Semester

2012-12

Permalink

http://hdl.handle.net/2142/42124

Copyright and License Information

Copyright 2012 Yeonsook Yi

Owning Collections