End-user Searching Behavior:
Cranfield University, UK
© 1997 Simon Bains firstname.lastname@example.org, Cranfield University
The Katharine Sharp Review ISSN 1083-5261, No. 4, Winter 1997
Rapid technological developments in the higher-education sector have resulted in a greater
reliance on end-user searching. It is important that those conducting literature searches for
research purposes are making the most effective use of the databases. In order to measure
searching behavior, a number of methods can be employed, and these can be quantitative or
qualitative in nature. All techniques have merit, but the researcher must understand the theories
about knowledge upon which the various methodologies rest, and must take care to choose those
techniques that are most suited to the investigation.
- Rapid developments in technology have had such a great effect on communication and
information storage and retrieval that the late 20th century has been termed the "information
society" (Corrall, 1995, p. 37). Information is available in more formats and from more sources
than ever before, and the "electronic revolution" has had a massive effect on our ability to store
and retrieve it. It might also have a profound effect on the role of the librarian intermediary, as
more people become familiar with an ever increasing number of user-friendly databases.
Libraries will need to consider making policy decisions about the way in which end-user
searching is incorporated into its services. Research into the effects of end-user searching is
vital to inform these decisions.
END-USER SEARCHING IN ACADEMIA
- The pace of change in the academic sector, in particular, has been rapid. Increased pressures on
space, dwindling resources and the growing popularity of part-time and distance learning have
resulted in a heavier reliance on remote access and less on local holdings, with more and more
information being stored electronically on CD-ROM or remote online databases, accessed
externally via computer networks and modems. According to Tenopir and Neufang (1995), a
survey of academic libraries in the United States revealed that 33% owned one hundred or more
CD-ROM titles, and 75% had the Local Area Network (LAN) capability to network CD-ROMs.
They also found that electronic information provision in the period 1990-1994 had risen
significantly (p. 69).
- The Follett Report (Joint Funding Council's Libraries Review Group, 1993) has predicted
large-scale change for higher education in the UK, particularly in the areas of electronic
publishing, networking and information systems. For the higher education sector, implications
include access to information from outside the library and a move toward end-user searching and
away from mediated searches. Before libraries adopt a policy of encouraging end-user
searching, however, librarians must be confident that such a move will not have a detrimental
effect upon the quality of the service. Conkling and Osif (1994) suggest that "CD-ROMs have
ushered in a new era in which the user is beginning to take responsibility for answering his own
information needs electronically" (p. 71). If this is so, then it is the responsibility of the
academic librarian to ask what effects such an approach will have on the research of those who
wish to adopt it. In order to answer this question, it is necessary to understand the various ways
that database usage can be measured.
- It has been suggested that "for many librarians, assessment is limited to rote counting of
anything and everything that is amenable to counting" (Sandler, 1992, p.184). Research in
librarianship has followed the general trend in the social sciences to place value on the
quantitative paradigm of the natural sciences. Recently, however, the belief of the qualitative
researcher that human behavior cannot be measured in this way has received greater respect.
- The dichotomy of quantitative and qualitative research has arisen because the manner in which
knowledge about the world is acquired has been understood within one of two paradigms. The
dominant natural science paradigm (Patton, 1991) has traditionally guided researchers in the
social sciences. This approach necessitates that the researcher believes the positivist view that
fundamental reality lies in physical objects and processes. Thus, human behavior is operating
according to cognitive rules or as a result of mechanical forces. The positivist believes that
knowledge can be guided by observable facts and that human behavior is sufficiently uniform to
be able to make accurate forecasts on the basis of statistically representative samples. Thus,
universal statements of truth can be made from quantitative data collected about cause and effect
relationships. Data is subjected to statistical analysis in order to test pre-conceived hypotheses.
- Qualitative research has developed as researchers have found that "[t]his use of quantitative
analysis and mathematical formulae in the natural sciences is not possible in the social sciences
because the concepts found in social theories can only be measured qualitatively" (Blaikie,
1993, p. 19).
- The qualitative researcher employs the phenomenological approach that naturally occurring
activity is worthy of investigation. This can be distinguished from the positivist belief in the
importance of certainty, which suggests that investigation should be carefully structured and
conducted under controlled conditions. Qualitative research is also concerned with the primacy
of the mind as the fundamental stimulus for human conduct. This means that primary data is
generated by the person under investigation, rather than empirical data derived from observable
processes. Patton (1991) describes this approach as the "representational theory of the mind" (p.
391). Knowledge is a representation of the world which has been cognitively worked by the
mind, and physical reality is, therefore, only known by an act of mind on phenomena presented
- Qualitative investigation is concerned not with cause and effect but with understanding, so
inquiry is a never-ending process as multiple interpretations are possible, based on individuals
and contexts. Instead of testing hypotheses, theories emerge as the data is recorded. Advocates
of grounded theory maintain that pre-conceived methods of data collection ignore the contextual
relevance of behavior and therefore remove its relevance. Instead, grounding the theory in the
data is advocated (Denzin & Lincoln, 1994, p. 204). This process, known as grounded theory, is
described by Weingand (1993) as "[i]nstead of beginning with a hypothesis that must be proven
either true or false, casting a broad net for data allows the theory to emerge over the process of
data collection--grounding the developing theory in the data itself" (p. 20).
- It is not necessary to adhere to one of the above paradigms to the exclusion of the other.
Methodological pluralism, or diversity, has the advantage of providing data which is
complementary, strengthening the likelihood of valid and reliable results (Wildemuth, 1993;
Patton, 1991). The important concern is to ensure that methods are employed flexibly so that
problem solving is effective (Weingand, 1993). However, logically incompatible findings will
result from mixing paradigms without conscious awareness that one is doing so (Patton, 1991).
It is also possible to use more than one method of data collection, a technique known as
methodological triangulation, in order to overcome any deficiencies in method (Fidel, 1993). It
is important to maintain an awareness of which approaches are being used to ensure that results
- The decision about qualitative or quantitative research techniques depends upon the ontological
and epistemological assumptions of the researcher and goals of the study. In other words, we
should ask what it is we are seeking to know and how such knowledge might best be acquired.
The question posed will determine which method or methods are appropriate for collection, and
these methods will then govern the nature of the data itself.
METHODOLOGICAL APPROACHES IN END-USER STUDIES
- Ephraim (1994) believes that the reliance on quantitative methods in librarianship arose from its
popularity in the social sciences, with which library science wished to be associated (p. 237).
We are warned that statistics are susceptible to misleading presentation and interpretation and a
carefully made judgement can be just as reliable. Sturges (1995) suggests a need to move
beyond enumeration towards understanding.
- Recent investigations into end-user searching have made use of the grounded theory technique of
qualitative research (Ellis, 1993; Wildemuth, 1993). Data is descriptive, and the process is
iterative, with tentative hypotheses being continually checked against the data and revised
accordingly (Bradley, 1993; Heyink & Tymstra, 1993). Thus, components are derived from the
data itself, rather than trying to force the data into preconceived, rigid structures in order to
make it quantifiable (Weingand, 1993, p. 19). Samples used in qualitative research can be
distinguished from the probabilistic sampling used in quantitative research. A small sample is
chosen, based on appropriateness to the topic. Bradley (1993) suggests that selection is made of
samples that are extreme, information-rich, critical or typical cases. The qualitative approach
can be seen as appropriate for an end-user study as these methods "tend to give more attention to
the subjective aspects of human experience and behavior" (Glazier & Powell, 1992, p. xi).
- Ellis (1993) advocates searching for categories and the consistent relation of incidents to those
categories, allowing their refinement and integration until several core categories remain, a
process he describes as "analytical exhaustivity" (p. 478). However, the researcher should
approach qualitative research with caution. Ellis argues that this process of deriving categories
and properties and demonstrating their relationship is "the most creative and intellectually
demanding part of the researcher's task" (p. 479).
- Wildemuth (1993) used "methodological pluralism" in a study aimed at discovering the
relationship between online searching and domain knowledge. The number of moves made and
their frequency was calculated, and protocols were used to test the hypothesis that regular use of
the 'repeat' key was related to accidental exits from the display. Wildemuth concluded that it is
possible to combine the generalisability of quantitative research and the detail of qualitative
techniques so that "the two approaches richly inform each other" (p. 466).
- Hancock-Beaulieu, Fieldhouse, and Do (1995), in an investigation into interactive query
expansion, use qualitative data derived from questionnaires to complement the quantitative data
collected from transaction logs. Su (1991) suggests that quantitative data relating to the length
of a search and number of documents retrieved can be usefully combined with qualitative data
relating to the satisfaction of the user with the search.
- Fidel (1993) argues that librarianship, although a social science, should "develop a
methodological framework and a set of procedures adequate for its own research" (p. 234).
- Whether the social science researcher chooses a quantitative or qualitative methodology, there
are three basic techniques available to collect the data: 1) interviews (both face to face and via
questionnaires); 2) observation; and 3) record examination.
- It is important that the technique or techniques chosen are carefully considered to ensure
maximum validity and reliability (Mason, 1994, p. 104; Altheide & Johnson, 1994). The
validity of a technique is its ability to provide an accurate measure of a concept which is relevant
to the hypothesis being tested. For it to be reliable, it must be consistent so that any change
should be the result of a change in the characteristic being measured, not the result of an
unreliable measuring instrument. This is also referred to as "test-re-test reliability" (Bailey
1987, p. 72). This is particularly important if the research is expected to be large-scale,
involving more than one researcher.
- An analysis of previous research in end-user searching has shown that questionnaires as a
method of data collection have proven consistently popular. The questionnaire offers subjective
data, which can be used to assess what people perceive to be the case. This can be a useful
approach if the perception of the user is an integral part of the research. Ankeny (1991), for
example, undertook research to ascertain whether user satisfaction equaled success. To this end,
the questionnaire proved a useful tool, especially as a sample of 600 users prohibited the use of
interviews. Hancock-Beaulieu et al. (1995) used the questionnaire to gain qualitative data to
complement quantitative data from transaction logs, an approach which enabled research to
evaluate system/user interaction (p. 235). Hsieh-Yee (1990) also combined quantitative and
qualitative methods in order to be sure that the most reliable data was gained, but argued that
while questionnaires and interviews are an effective way of gathering information about user
satisfaction and impressions, these methods will not allow any measurement of system
interaction. Su (1991) was interested in designing a model to help improve system development,
and found that the questionnaire could play a part in a quantitative/qualitative study. Ensor
(1992) agrees that the questionnaire is not necessarily accurate, but is the only way short of
interviews to elicit characteristics and subjective opinions. Interviews may be unfeasible due to
a large sample size.
- If at all possible, interviews should be carried out, as they offer a number of advantages over the
questionnaire. They are flexible, ideas can be followed up, and reactions probed further. The
response rate tends to be higher and more complexity is possible. (Bailey, 1987). According to
Sandler (1992), "open-ended questions by trained and neutral interviewers can determine what
the interviewee considers important" (p. 182). Denzin and Lincoln (1994) argue that "the
interview is the favourite methodological tool of the qualitative researcher" (p. 353).
- Dyer and Bouchet (1995) used interviews to discover users' opinions about the relevance of
references received. Questionnaires were also employed, in order to gain basic demographic
data and measure how users evaluated the information that was retrieved. Su (1991) made use
of the interview to try and eliminate some of the problems encountered in questionnaire-based
research, such as ambiguous wording, ill-defined categories and inadequate measuring
- According to Ellis (1993, p. 471), the ideal or root method of research is observation. Bailey
(1987) argues that it is the primary technique for the collection of non-verbal data and Bell
(1993) suggests that this method reveals characteristics which are impossible to discover by
other means. Certainly, if the area of interest is what people do, rather than what they say they
do, or if there are reasons to believe that the one will differ from the other, then observation is a
research method worthy of careful consideration. Bailey (1987) is of the opinion that,
[i]f a researcher is interested in gathering data on human actions, as oppose to
beliefs, values, or opinions, direct observation of the act by the researcher would
seem to have superior face validity over data collected by questionnaire . . . and
document study. (p. 265)
- However, there are drawbacks to observational studies regarding reliability. Quantification of
data can be complex (Sandler, 1992, p. 180), and it is debatable how consistently two
researchers will observe the same phenomena. Adler and Adler (1994) maintain that validity is
brought into question as researchers have to rely on their own subjective interpretation. Also, it
can be argued that a subject aware that he or she is under observation will act differently
(Mullings, 1984). They might feel embarrassment or the urge to 'show-off,' both of which
would have adverse effects on the reliability of the data. A potential solution would be to
observe anonymously, but this might be difficult if the observation study was lengthy or
complex. Conversely, Anderson (1995) argues "that most of the subjects did not find [the
presence of the researcher] of particular interest once they were aware that [the] sole purpose
was to gather data" (p. 367).
- Mullings (1984) comments that observation has been an under-used technique in librarianship,
but argues that "observation could play a useful part in library research, especially if used in
conjunction with other methods" (pp. 7-8).
- At the same time, it is acknowledged that care is needed when deciding how to approach this
method of investigation. Decisions have to be made about whether the study is to be overt or
covert, structured or unstructured, direct or indirect, in order to be satisfied that data yielded will
- Observation will allow the researcher to record what is actually happening, but it will not
necessarily reveal why it happens. For this reason many researchers have encouraged subjects to
think aloud, which can be used to assist in the recording of quantitative data or transcribed in
full and subjected to a qualitative analysis. Dalrymple and Zweizig (1992) used transcripts of
such "think aloud protocols" to measure reformulation behavior. This data was combined with
questionnaire results which were analyzed quantitatively. This is a user- rather than a
system-based analysis which, they suggest, is better suited to research on user behavior (pp.
177-178). Hsieh-Yee (1990) suggests that system- and user-based research be combined, using
protocols in association with transaction logs. A potential problem with protocols is that they
could endanger the validity of the data. Su (1991) suggests that video or audio recording of
protocols may affect the actions of some users, and dissuade others from taking part entirely (p.
- This is a method concerned with the examination of data in written form. For the purposes of a
study of information retrieval, this would involve an examination of the actual data retrieved
from the system. It is approached in a manner similar to an observation study, determining
variables and establishing a method for measuring them. Again, it is vitally important that care
is exercised in order to ensure that data is reliable.
- Record analysis is a social research method which is available to, and perhaps vital to, the
researcher into end-user studies. Conclusions can be drawn from an analysis of the search
strategy or the actual information retrieved and many systems make it possible to record
searches over a period of time (transaction logs). Researchers into end-user searching on library
OPACs have used this method (Ballard, 1994). Bysouth (1990) found logs useful to record the
searches of research scientists, but recognized that "we needed to acquire not only the details of
actual queries executed and strategies employed but also the user response to the various
selected approaches together with their level of satisfaction and the results obtained" (p. 111).
- According to Sandore (1993), the transaction log offers the advantage of allowing unobtrusive
observation. Hsieh-Yee (1990), however, argues that transaction logs, while unobtrusive and
facilitating ease of data collection, are limited in advancing our understanding of thought
processes, and Mitev, Venner, and Walker (1985) agree that "there is much that cannot be
deduced from even the most detailed transaction logs" (p. 176).
- Research has also used transaction logs to evaluate searching of full-text databases (Nicholas,
1995), but it can be argued that this method lends itself more to simple searches which are
designed to retrieve a particular piece of information, rather than the measurement of serendipity
and iterative techniques which are employed in the use of full-text databases. For example,
Wildemuth, Jacobs, and Fullington (1991) employed this method during research into the use of
a database of facts and concepts in microbiology: "To interpret the results, it is important to
remember that searchers were trying to identify one particular record in the database that would
satisfy each assigned problem" (p. 310).
- Ford, Wood, and Walsh (1994) employed content analysis in a qualitative manner, by asking the
student to judge the relevance of the documents retrieved. This was complemented by the
well-established quantitative techniques of precision and recall.
- Finally, any or all of the above methods can be incorporated into a comparative study, where the
aim is to compare one phenomenon with another. In the case of end-user searching, it is
obviously appropriate to consider how the searching behavior and results of the student compare
to those of a professional librarian. Several studies have made use of this technique (Ballard,
1994; Hsieh-Yee, 1990; Marchionini, Dwiggins, & Katz, 1993).
- The above study of the literature suggests that a combination of methods in order to maximize
the results is beneficial. The final decision on a methodological framework should be based on
- The epistemological and ontological beliefs of the researcher. Methods employed should
be in harmony with the researcher's own views about the sources of knowledge and the
manner in which this knowledge may be discovered.
- The epistemological and ontological goals of the study. The relative merits of positivist or
phenomenological approaches are dependent on the type of data required and the subject
being studied. Reasons for human behavior might best be determined by qualitative methods,
whereas a comparison of the performance of two computer systems using predetermined
variables could be carried out quantitatively.
- The size of the study. Even the most committed of qualitative researchers must admit that
very large studies do not lend themselves to such an approach. The amount of data generated
can be vast and the analysis time-consuming. The researcher should be prepared to be
flexible and adopt a combination of methods that provide the best compromise when faced
with a conflict between desired research methods and practical necessities.
- The final decision must rest with the researcher who needs to consider the advantages and
disadvantages of the different approaches, bearing in mind the possible benefits of
methodological pluralism. The need to ensure the success of the particular piece of research
should be regarded as of prime importance, and not the sociological and philosophical
arguments about methodological paradigms.
Adler, P. A., & Adler, P. (1994). Observational techniques. In N. K. Denzin & Y. S. Lincoln
(Eds.), Handbook of qualitative research. Thousand Oaks, CA: Sage Publications.
Altheide, D. L., & Johnson, J. M. (1994). Criteria for assessing interpretive validity in qualitative
research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 485-499).Thousand Oaks, CA: Sage Publications.
Anderson, J. (1995). Have users changed their style?: A survey of CD-ROM vs. OPAC product
usage. RQ, 34(3), 362-368.
Ankeny, M. L. (1991). Evaluating end-user services. Journal of Academic Librarianship, 16(6),
Bailey, K. D. (1987). Methods of social research (3rd ed.). New York: The Free Press.
Ballard, T. (1994). Comparative searching styles of patrons and staff. Library Resources and
Technical Services, 38(3), 293-305.
Bell, J. (1993). Doing your research project: A guide for first-time researchers in education and
social science (2nd ed.). Philadelphia, PA: Open University Press.
Blaikie, N. (1993). Approaches to social enquiry. Cambridge, UK: Polity Press.
Bradley, J. (1993). Methodological issues and practices in qualitative research. Library
Quarterly, 63(4), 431-449.
Bysouth, P. T. (1990). Evaluating the use of several approaches to online literature retrieval by
research scientists. In P. T. Bysouth (Ed.), End-user searching: The effective gateway to
published information (pp. 105-123). London: ASLIB.
Conkling, T. W., & Osif, B. A. (1994). CD-ROM and changing research patterns. Online, 18(3),
Corral, S. (1995). Academic libraries in the information society. New Library World, 96(1120),
Dalrymple, P. W., & Zweizig, D. L. (1992). Users' experience of information retrieval systems:
An exploration of the relationship between search experience and effective measures. Library
and Information Science Research, 14(2), 167-181.
Denzin, N. K., & Lincoln, Y. S., (Eds.). (1994). Handbook of qualitative research. Thousand
Oaks, CA: Sage Publications.
Dyer, H., & Bouchet, M-L. (1995). A comparison between the perceived value of information
retrieved via end-user searching of CD-ROMs and mediated online searching (BLRD Report
6208). British Library.
Ellis, D. (1993). Modeling the information seeking patterns of academic researchers: A
grounded theory approach. Library Quarterly, 63(4), 470-484.
Ensor, P. (1992). User practices in keyword and boolean searching on an online public access
catalog. Information Technology and Libraries, 11(3), 210-219.
Ephraim, P. E. (1994). A review of qualitative and quantitative measures in collection analysis.
The Electronic Library, 12(4), 237-241.
Fidel, R. (1993). Qualitative methods in information retrieval research. Library and Information
Science Research, 15(3), 219-247.
Ford, N.; Wood, F.; & Walsh, C. (1994). Cognitive styles and searching. Online and CDROM
Review, 18(2), 79-86.
Glazier, J. D., & Powell, R. R. (1992). Qualitative research in information management.
Englewood, CO: Libraries Unlimited.
Hancock-Beaulieu, M.; Fieldhouse, M.; & Do, T. (1995). An evaluation of interactive query
expansion in an online library catalogue with a graphical user interface. Journal of
Documentation, 51(3), 225-243.
Heyink, J. W., & Tymstra, T. (1993). The function of qualitative research. Social Indicators
Research, 29(3), 291-305.
Hsieh-Yee, I. P. Y. (1990). The search tactics of novice and experienced searchers: A
comparative study. University of Wisconsin-Madison.
Joint Funding Council's Libraries Review Group. (1993). Report [Follett Report]. (Chairman:
Professor Sir Brian Follett), The Councils.
Marchionini, G.; Dwiggins, S. S.; & Katz, A. (1993). Information seeking in full-text end-user
oriented search systems: The roles of domain and search expertise. Library and Information
Science Research, 15(1), 35-69.
Mason, J. (1994). Linking qualitative and quantitative data analysis. In A. Bryman & R. G.
Burgess (Eds.), Analyzing qualitative data (pp. 89-110). London: Routledge.
Mitev, N. N.; & Venner, G. M.; & Walker, S. (1985). Designing an online public access
catalogue: Okapi, a catalogue on a local area network (Library and Information Research
Report 39). The British Library.
Mullings, C. (1984). Observation. Centre for Research in User Studies, University of Sheffield.
Nicholas, D. (1995). Are information professionals really better online searchers than end-users?
In 19th National Online Information Meeting Proceedings (pp. 383-397). Learned Information
Patton, M. J. (1991). Qualitative research on college students: Philosophical and methodological
comparisons with the quantitative approach. Journal of College Student Development, 32(5),
Sandler, M. (1992). Qualitative research methods in library decision making. In J. D. Glazier &
R. R. Powell (Eds.), Qualitative research in information management. Englewood, CO: Libraries
Sandore, B. (1993). Applying the results of transaction log analysis. Library Hi Tech, 11(2),
Sturges, P. (1995). Field research in information studies: A Malawian study. In Proceedings of
the 1st British-Nordic LIS Conference, Copenhagen, 22-24 May, 1995. Royal School of
Librarianship, Denmark. Available from <
Su, L. T. (1991). An investigation to find appropriate measures for evaluating interactive
information retrieval. Unpublished doctoral dissertation. Rutgers, State University of New
Tenopir, C., & Neufang, R. (1995). Electronic reference options: Tracking the changes. Online,
Weingand, D. E. (1993). Grounded theory and qualitative methodology. IFLA Journal, 19(1),
Wildemuth, B. M. (1993). Post-positivist research: Two examples of methodological pluralism.
Library Quarterly, 63(4), 450-468.
Wildemuth, B. M.; Jacobs, E. K.; & Fullington, A. (1991). A detailed analysis of end-user search
behaviors. In Proceedings of the ASIS Annual Meeting, 28, 302-312.