Breakout Session
Monday, October 28, 11:30-12:30PM

Creating On-line Monitoring Log Reports to Support User Interviews

Presented by

Robert R. Downs
Stevens Institute of Technology
rdowns@attila.stevens-tech.edu

Reports are being created from on-line monitoring logs to support interviews of scholars who are using the Taylor archive. The interviews are being conducted currently as part of a study that is designing and evaluating an on-line research library being developed for use by individuals engaging in historical research. The study is part of the F. W. Taylor Project, which is digitally archiving papers from the Frederick Winslow Taylor Collection at the Stevens Institute of Technology.

Taking an iterative design approach, the study began by observing users of the special collection to determine the requirements for supporting the use of a digital version of the collection. Initially, semi-formal interviews of the patrons of the special collection were conducted, using the critical incident technique, to understand how scholars used the resources in the physical collection to conduct historical research.

Analysis of the interviews resulted in a model of the scholars' behavior while exploring the documents in the collection. The model of scholarly exploration behavior includes query composition, index searching, and contextual information processing activities. The initial functional and user interface design requirements for the on-line research library were based on the observations and the modeled behavior of the observed scholars.

The functional design requirements for the on-line research library include support for remote access to a single database of both page images and searchable document representations from the collection. The user interface design requirements include support for interactively querying the archive using natural language composition, related term selection, and related document selection features. In addition, logging capabilities are required to support user-centered studies on the use of the archive.

A prototype of the Taylor archive, meeting the initial design specifications, is currently being used by individuals who are conducting historical research remotely while participating in a field study investigating the use of the archive.

A procedure for participating in the study was developed and pre-usage and post-usage survey questionnaires were designed, based on previous studies, to measure the user attributes of the participants. The participation procedure, including the questionnaires and a tutorial on using the archive, also has been placed on-line.

Individuals who have a scholarly interest in Taylor, the history of management, and the history of human factors are being asked to use the archive and to participate in the study investigating the use of the archive. Besides completing the pre-usage and post-usage questionnaires, participants are being asked to maintain a list of the dates and times when they use the archive and to be interviewed by telephone on their use of the archive.

The on-line log report is created for the participating user being interviewed and only contains information that has meaning to that user. Each participant's self-reported dates and times of searching the archive are matched with records of server activities within the access logs and audit logs from the web server hosting the archive. These records are used to create a sequential report containing the dates and times of on-line activities and the titles of documents accessed by a participant while using the archive.

A draft of the log report is sent to the participant for verification and correction, if necessary. The completed log report is presented to the participant to be referred to during the interview and a telephone interview of the participant is scheduled.

An interview guide also has been developed and is consulted by the interviewer during the interviews. However, the on-line log report has become the primary instrument to support the dialogue between the interviewing researcher and the responding participant.

Even though it is difficult to convince those who use the archive to participate in a post-usage interview, the data obtained from the interview is a valuable source of information, providing much insight on the experiences of the individual searchers. Besides supporting the interview process, the log reports are facilitating research on the use of the archive by providing a sequential record of participants' on-line activities and by allowing participants to confirm and correct the accuracy of this record.

Log reports are supporting the interview process by providing cues to the interviewer for directing the interview, by stimulating respondents' recollections of their on-line behavior, and by providing a reference for discussing and reflecting on this behavior. The log report also facilitates discussions of how the questions, the goals, and the strategies of the respondent have changed during the course of their on-line research.

In addition to supporting discussions of a respondent's sequential activities, the log report supports discussions of critical incidents. Discussions of any critical incidents experienced by the respondents are initiated, during the interview, after the respondent's sequential activities have been discussed.

Finally, the use of the on-line log reports can be adapted to a diverse population of users who have varying experiences, either with the content of the archive or with the use of on-line learning environments.

The activities presented have been taken in part from a dissertation to be submitted in partial fulfillment of the requirements for the Ph.D. degree of Doctor of Philosophy in Information Management in the Department of Management and Engineering Management of the Stevens Institute of Technology.

Example of log report entries:

Taylor Archive August 30, 1996
7:05 to 8:05 a.m. PDT - Approximately 10:05 to 11:05 a.m. EDT

10:25:38 The Taylor System of Shop Management at the Atertown Arsenal
10:45:16 Contents
10:52:28 Frederick W. Taylor

Special thanks to Paul Doty for recording notes during the presentation.

Last Updated: Nov. 13, 1996