Robert R. Downs
Doctoral Student of Information Management
and Designer F. W. Taylor Project
S. C. Williams Library
Stevens Institute of Technology
Castle Point on Hudson
Hoboken, NJ 07030
(201) 655-7025
(201) 655-5455 fax
rdowns@stevens-tech.edu
http://attila.stevens-tech.edu/~rdowns
The Taylor Archive
The prototype of the on-line Taylor archive includes the features
established from the initial design requirements study that began when the
Taylor Project was initiated. The prototype implementation contains the
complete text and the page images of over seven thousand pages from the
F. W. Taylor Collection. The complete text of the represented documents
has been indexed utilizing natural language processing and is searchable
using natural language query features, including query composition,
related term selection, and related document selection.
For each of the documents retrieved by a query, a descriptive record is displayed as a label to support selection for viewing by the user. Each document is the smallest unit of either an article, a chapter, or a section from the books represented. Documents can be viewed as individual page images and viewed as scrollable text documents that have resulted from the optical character recognition process.
Currently, the prototype of the on-line Taylor archive is being used by scholars conducting historical research on the represented domain of Scientific Management while the scholars' use of the archive is being observed as part of a field study investigating the on-line inquiry behavior of historical researchers. Each of the Taylor Project studies, including the current study, has focused on the users and has been conducted as part of the effort to design and evaluate an on-line research library.
The Taylor Project User Studies
The Taylor Project user studies have been conducted to design and evaluate
the on-line Taylor archive, a dual purpose digital library. The on-line
Taylor archive both serves as an on-line research library to support
scholarly inquiry in the represented domain of Scientific Management and
serves as a field laboratory for studying the use of an on-line research
library.
Besides the study investigating the initial design requirements for an on-line research library and the study currently being conducted to investigate the inquiry behavior of historical researchers using an on-line research library, three preliminary field studies also have been conducted to examine the use of on-line library resources.
Three of the studies, the initial design requirements study, a classroom laboratory observation study, and the field study currently being conducted on the use of an on-line research library, have employed naturalistic methods to investigate the inquiry behavior of individuals conducting historical research. On the contrary, two other studies, a bibliographic database user study and a study investigating students' expectations for using an on-line database, were conducted as survey investigations to identify measures that might be utilized in successive studies of users' on-line inquiry behavior.
The methodologies employed for each of the five studies conducted as part of the Taylor Project will be briefly described in the order in which the studies have been conducted.
The first user study of the Taylor Project was conducted to establish the initial design requirements for the on-line Taylor archive. Adopting a behavioral approach to the design of an information retrieval system, similar to the model advocated by Ellis (1989), the design requirements were investigated during a field study that observed current and recent patrons of the F. W. Taylor Collection on their use of the physical resources located in the special collection. The initial design requirements study utilized the grounded theory methodology advocated by Strauss and Corbin (1990), for data collection and analysis.
Based on early observations and informal interviews of the scholars currently using the Taylor Collection, an interview guide was developed to conduct semi-structured telephone interviews of the Taylor scholars. The interviews were conducted to elicit participating scholars' perceptions of their use of the Taylor Collection. Flanagan's (1956) Critical Incident Technique was employed during the interviews to stimulate and elicit reports of research experiences from the respondents. The telephone interview sessions were recorded on audio tape and transcribed for analysis. The interview transcripts were analyzed using the constant comparison method. The findings of the initial design requirements study resulted in the establishment of functional and user interface design specifications for an on-line research library (Downs,1995).
The first survey study that was conducted to investigate potential measures for subsequent studies was a longitudinal field investigation of the relationships between self-reported perceptions of use, self-reported learning styles, self-reported previous experience, and instructor-reported performance of end-users. The subjects of this study were university students who were using a bibliographic database to complete a library research assignment for an undergraduate Psychology course.
The study was conducted by administering pre-usage and post-usage survey instruments to the students before and after they had completed their research assignment using the bibliographic database. Student survey responses were compared statistically, along with the students' scores on their library research assignments, as graded and reported by the course instructor, to determine the relationships between the variables studied.
The second survey-based field study investigated the relationships between self-reported expectations of using a full-text database to conduct historical research and previous experience. The subjects of this investigation were students who were using printed resources to conduct library research on Taylor and Scientific Management to complete their library research assignment for a graduate level Total Quality Management Course. A single questionnaire was administered to the students in class before they received their research assignment. The variables that were measured for this study included self-reported expectations of using a full-text database to complete their library research assignment, self-reported domain and system experience, and self-reported learning style. The subjects' survey responses were analyzed statistically.
The next preliminary field study of the Taylor Project utilized the participant observation method to observe students' on-line inquiry behavior while they were using both web resources and a full-text database to complete assigned library research on Galileo Galilei for an undergraduate History of Science course. The students were observed on three occasions as they performed on-line library research in a classroom computer laboratory where they met to work individually and in groups on their research assignments during scheduled course meeting times.
While observing the students who were engaged in completing their research assignments, the investigator interviewed each of the students in succession, at various time intervals, to obtain reports from the students on their library research activities, their experiences, and their progress. The investigator took field notes on what was observed and reported by the students. After each session, the field notes from each of the observed sessions were analyzed, using the constant comparison method, to obtain an understanding of the process in which the users were engaging.
Currently, individuals who have actual information needs for conducting research on the represented domain of Scientific Management are being requested to participate in a study on the use of the on-line Taylor archive. The primary research question for the current study asks how historical researchers learn while using natural language query tools to explore the contents of a full-text and multimedia on-line research library.
The current investigation is taking a case-study approach to observe how on-line library researchers utilize the functional and the user interface features of an on-line research library to answer their information-seeking questions. An international group of scholars who are conducting historical research using the on-line Taylor archive are currently participating in the study by completing a procedure for participation while they use the archive to conduct their research.
Each participant is being requested to complete a tutorial on using the archive before their initial use of the archive. Data collection efforts for the observation of participants includes administering a pre-usage and a post-usage survey to each participant, monitoring participants' on-line activities, requesting a self-reported list of the dates and times each participant used the archive, and interviewing participants about their on-line behavior. Participants also are encouraged to send electronic mail to the principal investigator stating any comments or questions that they might have on the use of the archive.
The procedure for participation in the current study, a description of the archive, the survey instruments, and a tutorial for using the archive that provides access to the archive are administered on-line (Taylor Project, 1996). Scholars interested in studying representations of the original documents about Scientific Management that are written by Taylor and his colleagues are being requested to participate in the study by completing the requirements for participation and by using the on-line research library to conduct their research.
Allerton Institute Influence
The methodologies employed for the participant observation study and for
the current case-based study of on-line library researchers' use of the
Taylor archive have been considerably influenced by presentations and
discussions that took place during the Thirty-Seventh Annual Allerton
Institute (1995). For example, both of these investigations utilize
naturalistic inquiry methods to observe the evolving information needs,
inquiry behavior, and perceptions of end-users who are engaging in actual
historical research while using an on-line research library. Also, the
current investigation is utilizing the micro-moment time-line approach
(Dervin, 1995) to elicit recollections of experiences from participants
during the interviews.
Recommendations for Allerton '96
It is anticipated that participating in Allerton '96 will provide
opportunities to learn more about similar naturalistic and case-study
methodological approaches and techniques used and planned by other
researchers who are studying digital library users. In addition, benefits
to other researchers might result from this author reporting about the
studies previously conducted and mentioned in this report, reporting on
the current status of the study being conducted, and describing the
methods that are being used and planned for the current study.
Receiving comments, criticism, and advice about the current study, from the other researchers participating in Allerton '96, also could be beneficial to this investior. Specific suggestions from others for the analysis of data being collected for the current study would be appreciated very much. For example, participant interviews will yield large transcripts that must be analyzed. Techniques that others might share for analyzing such transcripts could be extremely helpful. Furthermore, on-line monitoring log files also will represent volumes of data that must be organized and analyzed. Any suggestions that other researchers might have for organizing and analyzing the on-line monitoring data would be inspiring.
In addition, other participants might benefit from a presentation of the issues and details of the methodology being used for the current case-based study on the behavior of on-line library researchers, previously mentioned. The case-study method being utilized in the current study could be compared with naturalistic and case-study methods being employed by other researchers, as well. Alternatively, the case-study method currently being employed could be compared with the participant observation technique for observing users of digital library systems.
Also, a breakout session that explores preliminary problems encountered or anticipated methodological concerns might be useful for many researchers. In such a session, more experienced researchers might compare their problems or concerns with those of less experienced researchers, resulting in discussions about methodological procedures and techniques that might be employed to resolve outstanding issues.
Another suggestion would be to organize workshops to discuss the collection and analysis of specific types of data. For example, a workshop on collecting and analying on-line monitoring data might be helpful to many participants. Similarly, researchers might benefit from participating in a workshop where individuals share techniques for interviewing users about their on-line experiences. Techniques for utilizing and analyzing the data from various data collection activities, to facilitate the interview process, also could be discussed in such workshops. For example, in the current study previously discussed, the self-reported on-line diary logs and the on-line monitoring log reports of participants are used during the interviews to facilitate discussion about the respondents' research activities while they were using the on-line research library.
The activities reported in this document have been taken in part from a dissertation to be submitted in partial fulfillment of the requirements for the Ph.D. degree of Doctor of Philosophy in Information Management in the Department of Management and Engineering Management of the Stevens Institute of Technology.
Dervin, B. (1995). Chaos, Order, and Sense-Making: A Proposed Theory for Information Design. Thirty-Seventh Annual Allerton Institute. How We Do User-Centered Design and Evaluation of Digital Libraries: A Methodological Forum. Graduate School of Library and Information Science. University of Illinois at Urbana-Champaign. http://edfu.lis.uiuc.edu/allerton/95/s5/dervin.draft.html
Downs, R. R. (1995). The Frederick Winslow Taylor Project: User-Centered Design and Evaluation. How We Do User-Centered Design and Evaluation of Digital Libraries: A Methodological Forum. Graduate School of Library and Information Science. University of Illinois at Urbana-Champaign. http://edfu.lis.uiuc.edu/allerton/95/s3/downs.html
Ellis, D. (1989). A Behavioural Approach to Information Retrieval System Design. Journal of Documentation. 45(3), 171-121.
Flanagan, J. C. (1954). The Critical Incident Technique. Psychological Bulletin, 51(4), 327-358.
Strauss, A. & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage Publications.
Taylor Project (1996). http://attila.stevens-tech.edu/~rdowns
The Thirty-Seventh Annual Allerton Institute (1995). How We Do User-Centered Design and Evaluation of Digital Libraries: A Methodological Forum. Graduate School of Library and Information Science. University of Illinois at Urbana Champaign. http://edfu.lis.uiuc.edu/allerton/95/