The Alexandria Project will develop a user-friendly digital library system providing comprehensive services to catalog, browse and retrieve distributed collections of maps, images, and other spatially referenced information. One goal of the project is to eliminate the conventional library separation between text and 'special' archives. Alexandria will merge access to traditional archives (e.g.., books, maps and photos) with spatial data archives (e.g.., digital coordinates and imagery). A digital library must be more than a digital conversion of a conventional map library, and should reflect a fundamental rethinking of how spatial information is captured, stored, and accessed. In the context of spatial data, the distinction between catalog and materials, or between metadata and data should become transparent. Users will be able to search for maps and images on the basis of their content as well as by reference to spatial location.

Specific technical impediments to be overcome include: the lack of appropriate spatial indexes to accommodate a wide variety of queries: performance problems due to the large volumes characteristic of spatial information; the lack of an appropriate spatial metadata model; and a lack of understanding of user requirements, and an empirical basis for user interface design. The last of these creates the focus for my research in Alexandria. I head the User Evaluation Team for the Alexandria Project. My job includes determining user requirements for the Library, designing user evaluation test instruments to determine that these requirements are appropriate, and testing users empirically to ascertain if the current system meets those user requirements. The results of my research inform the User Interface Team and guide some of the system revisions.

This job presents several interesting challenges. First, we are developing multiple versions of the Library. At present we have testbeds running on UNIX and Windows platforms, and are completing our first Web version sometime this fall. Each version contains somewhat different functionality, and system differences require that we develop an evaluation paradigm that is flexible. One of our testbeds can accommodate interactive logging of user sessions, and we plan to analyze the logs using Protocol Analysis. WE are adapting the intercoder reliability checks to compare user logs for different system versions, for different classes of users, and to model system constraints such as network delays. I am interested to hear if others have applied Protocol Analysis in this way.

We tried think-aloud and talk-aloud protocols and find they don't work very well, since users tend to either use the interface and not talk, or talk and stop working with the system. To get around this we intend to videotape some sessions, and then ask users to talk-aloud as they view the videotape. Between the interactive logs, the videotape, the questionnaire, and the talk- aloud reports, the sheer volume of data that can be collected is mind- boggling, and I am interested in hearing from others how they reduce their data collection modes to those most informative to their evaluations.

We also find that navigating in the Library is not self-evident to first time users, and have designed tutorials as a result. We also use a descriptive questionnaire ("what did you like about the menu tools" and "did any icons confuse you" etc.). The tutorials work well for some users, but it is clear that patterns of motivated browsing and retrieval differ from use patterns generated in the tutorials. I haven't figured out how to create motivated browsing or retrieval scenarios, nor how to create a statistical referent to which I can anchor such scenarios, although I recognize this as a significant consideration. I am interested in hearing how other user evaluation researchers confront this issue.

Another challenge is narrowing the user classes we intend to test. From an original intention to test earth and space scientists, government spatial data producers, commercial and public sector spatial data users, academic researchers, K-12 students, and the library community, we have quickly realized the infinity of spatial data users out there. At present, we have decided to target specific library environments, and prioritize the spatial data users who patronize those environments. I am most interested in discussing the selection of target populations for user evaluation at the conference, since the users who are tested will in part dictate the testing venue.

Finally, it is clear that user requirements change in the course of using any version of the system. Thus user evaluation is effectively shooting at a moving target. To account for this we are instituting longitudinal experiments, returning a number of times to the same users. We hope to identify shape and steepness of learning curves for existing and revised versions of the system, and determine which interface tools transfer easily to other platforms.

I hope this gives readers a fair idea of what I hope to find out at the workshop. In terms of what I can contribute, I will bring a CD-ROM version of the Library to demonstrate what we are trying to do, and to encourage discussion about examples of working digital libraries.