|Title:||An approach to the evaluation of expert systems for the selection of reference sources|
|Doctoral Committee Chair(s):||Lancaster, F.W.|
|Doctoral Committee Member(s):||Smith, Linda C.; Allen, Bryce L.|
|Department / Program:||Library and Information Science|
|Discipline:||Library and Information Science|
|Degree Granting Institution:||University of Illinois at Urbana-Champaign|
|Abstract:||Sixty students at the University of Illinois voluntarily participated in the research to select reference sources for ten test questions with or without the aid of the expert systems to be evaluated. The students were first divided into little-experienced and no-experienced groups; the students in each group were further randomly divided into RE, SF, and NO groups. The RE group students consulted Reference Expert to select reference sources, the SF group students consulted SourceFinder, and the NO group students consulted the online catalog (IO$\sp+$/IBIS), browsed shelves in the Undergraduate Library, or retrieved from personal knowledge.
The results of the study suggest that the current expert systems for the selection of reference sources cannot perform as well as experienced subject-oriented reference librarians. The better system, Reference Expert, could only achieve 80 percent accuracy in aiding users to select appropriate sources. SourceFinder achieved 40 percent accuracy. A failure analysis is provided. The results indicate that the students without "expert system" aid can perform as well as comparable students aided by SourceFinder in the selection of sources. SourceFinder can lead its users to select reference sources at a 25% consistency level, while Reference Expert achieves a 57.5% consistency level. Students not aided by any expert system achieve only 10% consistency in source selection.
The results of this research generally do not offer strong support for the belief that "expert systems" for selection of reference sources can provide much help to increase the accuracy of question answering in reference services. In fact, in the case of SourceFinder, students of little experience aided by the system performed significantly worse than did the comparable students using the online catalog or personal knowledge. It is not just a matter of "why bother" to build an expert system of this type; there also exists the possibility that exclusive use of such a system to select reference sources may worsen the existing "half-right" performance of reference librarians.
|Rights Information:||Copyright 1994 Su, Shiao-Feng|
|Date Available in IDEALS:||2011-05-07|
|Identifier in Online Catalog:||AAI9512565|
Files in this item
|(no description provided)|