Files in this item



application/pdf8127718.pdf (6MB)Restricted to U of Illinois
(no description provided)PDF


Title:The Logic of Criterion-Referenced Testing
Author(s):Tomko, Thomas Norbert
Department / Program:Education
Degree Granting Institution:University of Illinois at Urbana-Champaign
Subject(s):Education, Tests and Measurements
Abstract:This thesis is a critical analysis of the currently dominant approach to criterion-referenced testing (CRT). The main point of the thesis is that the logic underlying the currently dominant CRT model cannot support the type of inferences desired in CRT. The logical model that underlies CRT is a statistical generalization model. Due to the nature of the elements of the populations sampled under such a model, the inferences that are desired from CRT cannot be supported by statistical generalization alone. It is argued that the inferences desired are inferences about abilities. Various interpretations of ability claims are examined and the problems with CRT theory are related to the broader dispute in philosophy of science between realism and nominalism.
The thesis consists of five chapters. Chapter I includes an introduction to and motivation for the thesis. In Chapter II, the important features of CRT theory are outlined. The broadest characterization of CRT on which there is general consensus is that CR interpretations tell one what an examinee can or cannot do. The main contention of Chapter II is that the dominant approach to CRT relies on a statistical generalization model for its underlying logic. Differences among CRT theorists as to the nature of the elements in a universe or domain in CRT are pointed out. The three main candidates for universe elements are behaviors, items, and observations.
In Chapter III the application of the statistical generalization model to CRT is examined in detail. Several problem areas in statistical generalization in general were noted. Foremost among these is the problem of procuring a representative sample. The most widely accepted method of sampling is random sampling which does not guarantee representativeness, but does give one some idea of the likelihood that a sample is representative. Populations that are subject to statistical generalization using random sampling must be well-defined and closed.
It is argued that two of the candidates for universe elements, behaviors and items, have a type-token ambiguity that must be kept in mind when applying a statistical generalization model. It is argued that behaviors, in either the type or token sense, and item types are not proper universe elements for CRT. It is argued that item-tokens (which can be considered a type of observation) might be appropriate as universe elements in the application of the statistical generalization model to CRT, if restricted in certain ways.
In Chapter IV it is argued that a statistical generalization model cannot support the type of inference desired in CRT. In order to show this type of inferences desired from CRT are first examined. These inferences are claims that individuals do or do not have certain abilities. Next the problem of how one establishers such claims is examined. It is argued that the evidence that one would accept as supporting an ability claim would depend on whether one held a nominalist or a realist view of abilities. It is then argued that the type of claims that are desired from CRT to some extent presuppose a realist view and that such claims cannot be supported by a statistical generalization model alone. To support such claims one needs a broad CRT model that allows inferences not strictly permitted by statistical generalization alone.
In Chapter V consequences for testing practices and implications for future research are discussed. It is argued that the type of inferences desired from CRT can be completely supported only by an inference-to-the-best-explanation model. Topics for future research are noted. These include the problem of specifying what constitutes an adequate explanation, the nature of causal claims, and the topic of natural kinds in the area of human abilities.
Issue Date:1981
Description:131 p.
Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 1981.
Other Identifier(s):(UMI)AAI8127718
Date Available in IDEALS:2014-12-12
Date Deposited:1981

This item appears in the following Collection(s)

Item Statistics