Thomas A. Finholt
University of Michigan
I gratefully acknowledge the assistance of Stephane Cote, Gayle Farbman, Joe Magee, Stephanie Mackie-Lewis, Susan McDaniel, William Mott, Anthony Scaglione, Katrina Wade, and Beth Yakel for data collection and analysis. I am also grateful to my fellow investigators on the UARC and Medical Collaboratory projects: Ron Adler, Dan Atkins, Bob Clauer, Michael Cohen, Farnim Jahanian, Chuck Meyer, Gary Olson, Craig Rasmussen, Atul Prakash, and Terry Weymouth.
Requests for further information should be sent to: a) Thomas Finholt, Department of Psychology, The University of Michigan, 525 East University, Ann Arbor, MI 48109-1109; or b) finholt@umich.edu
Collaboratories are computer-supported systems that allow people to work with each other, facilities, and databases without regard to geographical location. Interdisciplinary research at the University of Michigan is exploring the impact of collaboratories in two domains. The Upper Atmospheric Research Collaboratory (UARC) is designed to increase access to distant facilities and scientists in the space physics community. The Medical Collaboratory is designed to increase contact between geographically distributed primary care physicians and radiologists in consultations over radiograph and ultrasound images. The development strategy in both projects is user-centered. Behavioral scientists contribute to this effort through characterization of conditions prior to collaboratory introduction and through measurement of changes produced by collaboratory use. A similar strategy is proposed for analysis of digital library efforts.
The University of Michigan is a leader in the development of collaboratories. A collaboratory is the "...combination of technology, tools and infrastructure that allow scientists to work with remote facilities and each other as if they were colocated." (Lederberg & Uncapher, 1989, p. 6) A National Research Council (1993) report defines a collaboratory as a "...center without walls, in which the nation's researchers can perform their research without regard to geographical location -- interacting with colleagues, accessing instrumentation, sharing data and computational resources [and] accessing information in digital libraries." (National Research Council, 1993, p. 7) A simplified form of these definitions describes a collaboratory as the use of computing and communication technology to achieve the enhanced access to colleagues and instruments provided by a shared physical location, but in a domain where potential collaborations are not constrained by temporal or geographic barriers.
Currently there are two collaboratory efforts at the University of Michigan. The Upper Atmospheric Research Collaboratory (UARC), started in 1992, is an attempt to build a system to support science among an international community of space physicists focused on the Sondrestrom Upper Atmospheric Research Facility, located in Kangerlussuaq, Greenland (Clauer, 1994). The UARC interface provides data viewers that display real-time images of ionospheric phenomena, along with "chat windows" that allow scientists to discuss the displays by typing messages to one another. The UARC system became operational in April, 1993 and is now in use by physicists at eleven laboratories in North America and in Europe. When conceived, the principal motivations for the UARC system were concerns among space physicists that they were unable to respond rapidly to interesting ionospheric phenomena, that they did not have good opportunities to corroborate data from one instrument with data from other instruments, and that the end of the Cold War had eliminated low cost military flights to bases in Greenland near the Sondrestrom facility. In these terms, the UARC was envisioned as a system that would, as much as possible, re-create the experience of being in Greenland through use of the collaboratory technology.
The Medical Collaboratory, started in 1995, is an attempt to build on lessons learned from UARC to support consultations between geographically distributed primary care physicians and radiologists over radiograph and ultrasound images. As in UARC, the Medical Collaboratory interface will provide viewers that display radiological images with annotations, but with the addition of video and audio conferencing. The Medical Collaboratory system is not yet operational, but is envisioned to connect doctors at satellite clinics in the vicinity of Ann Arbor with radiologists at the University of Michigan Medical Center.
Both collaboratory projects at the University of Michigan are dedicated to a user-centered design approach. In practice this has several implications. First, in UARC, it has meant a commitment to develop software in programming environments that allow rapid prototyping (i.e., NeXTStep). This means that systems are deployed to users rapidly and designers and evaluators can receive feedback and make quick modifications. Second, in UARC, rapid development cycles have created demanding production schedules. Finally, by placing collaboratory technology at the core of practice in both space physics and radiology, collaboratory performance is critical and highly visible. In cases this has produced "success disasters, "where system development has not kept up with escalating user expectations. Additionally, growing dependence on collaboratories has led to greater allocation of research effort to testing and fixing existing systems, sometimes at the expense of producing the next generation of systems.
A consequence of user-centered design in the collaboratory projects is that system developers must be able to distinguish when a particular system or modification has positive impact on user work practices. A crucial part of obtaining this understanding is producing an accurate picture of how work is done prior to the introduction of technology. Therefore, in both UARC and the Medical Collaboratory, behavioral scientists work alongside computer scientists and users to analyze existing work practices, changes that occur in these practices through the use of technology, and the impact of these changes. An important component of this work is studying the effects of technology in the field. That is, a data collection strategy that captures what real people do with real technology, as opposed to laboratory studies that focus on artificial settings and tasks.
The belief, in both collaboratory projects, is that gains resulting from the introduction of technology are highly likely. The smooth introduction of this technology, however, is not a foregone conclusion. To improve the likelihood of success -- i.e., to increase the probability that a system will function productively in the full complexity of actual work contexts -- both projects strive to continuously monitor and refine deployed systems. The task for behavioral scientists, then, is to find fast, informative ways of understanding the actual work settings for which new information technology tools are being developed. The goal in a user-centered design effort is to inject these observations back into the design process to: a) provide a baseline for evaluating future changes; and b) to illuminate productive directions for prototype development.
Within the UARC and Medical Collaboratory projects behavioral scientists have contributed useful insights through the application of multiple data collection methods. First, both projects rely heavily on documenting work practices and experiences with technology through the use of video recordings. At an informal level, the recordings allow the behavioral scientists to illustrate observations via a portable medium. That is, system developers often can't be present to watch systems in use and video recordings provide a convenient means to store and share illuminating lapses or failings in deployed systems. At a formal level, the recordings represent a log of activity that can be coded and analyzed. Both collaboratory projects use semi-automatic annotation technology, such as CVideo, to speed the coding and analysis of videotape. A second data collection method used in both projects is extensive field observation combined with semi-focused interviews. With UARC, for example, these observations focus on the annual experimental campaigns mounted by the space scientists to coincide with ideal viewing periods during winter in the northern hemisphere. These observations provide a rich context for communicating user demands as well as for communicating the motivation for user demands. The UARC project uses a longitudinal survey to track user attitudes and behavior over the course of the project. The instrument assesses levels of system use, general communication behavior, scientific productivity, and patterns of contact among UARC users. Finally, both the UARC and Medical Collaboratory projects rely on automatic data gathering in the form of system logs. For instance, the UARC system tracks the number of logins per user, activities during sessions, and the content of the system chat window. These automated data are used to calibrate and validate self-reported data from users.
Digital libraries differ in significant ways from the collaboratory projects described above. Notably, while collaboratories are designed to meet the needs of a relatively circumscribed set of users, digital libraries must be accessible to thousands of users. Further, while collaboratory users may be assumed to possess certain critical similarities (e.g., UARC users are all space pysicists, Medical Collaboratory users are all doctors) digital library users are presumed to be more heterogeneous. This creates two difficulties for analysts of digital libraries. First, for purposes of sampling and instrumenting system use, who constitutes the relevant user sample for digital libraries? As an illusrtation of this difficulty at Michigan, the University of Michigan Digital Library (UMDL) aims to satisfy secondary students at one end of the continuum and experienced Ph.D.-level scientists at the other. Tracking all of these users poses significant difficulties. For example, insights gained from observing space science faculty might indicate a set of design considerations wholly inappropriate for less advanced users. Second, when users don't bring shared assumptions and expectations to a technology who or what arbitrates among conflicting demands? This concern is most obvious in the tension between content managers and content users. For example, on the part of managers, there is an understandable desire to minimize transactions with publishers and follow a "one stop shop" model of acquisition. However, this strategy does not necessarily satisfy or excite potential users. As an illustration, the current UMDL does not offer any of the most read journals reported by 67 space scientists at the University of Michigan and other leading research universities. In this situation, then, the behavioral scientist is confronted with the awkward complexities inherent in the idea of a digital library: Evaluation of a system may show a perfect match to user demands -- but low utilization due to factors external to the actual system design.
Clauer, R., Rasmussen, C.E., Niciejewski, R.J., Kileen, T.L., Kelly, J.D., Zambre, Y., Rosenberg, T.J., Stauning, P., Friis-Christensen, E., Mende, S.B., Weymouth, T.E., McDaniel, S.E., Olson, G.M., Finholt, T.A., & Atkins, D.E. (1994). New project to support scientific collaboration electronically. Eos, 75, June.
Lederberg, J., & Uncapher, K. (1989). Towards a national collaboratory: Report of an invitational workshop at the Rockefeller University. Washington, D.C.: National Science Foundation, Directorate for Computer and Information Science.
National Research Council. (1993). National collaboratories: Applying information technology for scientific research. Washington, D.C.: National Academy Press.