End-user Searching Behavior:
Considering Methodologies

Simon Bains
Cranfield University, UK


© 1997 Simon Bains sbains@cranfield.ac.uk, Cranfield University
The Katharine Sharp Review ISSN 1083-5261, No. 4, Winter 1997 [http://edfu.lis.uiuc.edu/review/winter1997/bains.html]

    Rapid technological developments in the higher-education sector have resulted in a greater reliance on end-user searching. It is important that those conducting literature searches for research purposes are making the most effective use of the databases. In order to measure searching behavior, a number of methods can be employed, and these can be quantitative or qualitative in nature. All techniques have merit, but the researcher must understand the theories about knowledge upon which the various methodologies rest, and must take care to choose those techniques that are most suited to the investigation.


    INTRODUCTION

  1. Rapid developments in technology have had such a great effect on communication and information storage and retrieval that the late 20th century has been termed the "information society" (Corrall, 1995, p. 37). Information is available in more formats and from more sources than ever before, and the "electronic revolution" has had a massive effect on our ability to store and retrieve it. It might also have a profound effect on the role of the librarian intermediary, as more people become familiar with an ever increasing number of user-friendly databases. Libraries will need to consider making policy decisions about the way in which end-user searching is incorporated into its services. Research into the effects of end-user searching is vital to inform these decisions.


    END-USER SEARCHING IN ACADEMIA

  2. The pace of change in the academic sector, in particular, has been rapid. Increased pressures on space, dwindling resources and the growing popularity of part-time and distance learning have resulted in a heavier reliance on remote access and less on local holdings, with more and more information being stored electronically on CD-ROM or remote online databases, accessed externally via computer networks and modems. According to Tenopir and Neufang (1995), a survey of academic libraries in the United States revealed that 33% owned one hundred or more CD-ROM titles, and 75% had the Local Area Network (LAN) capability to network CD-ROMs. They also found that electronic information provision in the period 1990-1994 had risen significantly (p. 69).

  3. The Follett Report (Joint Funding Council's Libraries Review Group, 1993) has predicted large-scale change for higher education in the UK, particularly in the areas of electronic publishing, networking and information systems. For the higher education sector, implications include access to information from outside the library and a move toward end-user searching and away from mediated searches. Before libraries adopt a policy of encouraging end-user searching, however, librarians must be confident that such a move will not have a detrimental effect upon the quality of the service. Conkling and Osif (1994) suggest that "CD-ROMs have ushered in a new era in which the user is beginning to take responsibility for answering his own information needs electronically" (p. 71). If this is so, then it is the responsibility of the academic librarian to ask what effects such an approach will have on the research of those who wish to adopt it. In order to answer this question, it is necessary to understand the various ways that database usage can be measured.


    METHODOLOGICAL ISSUES

  4. It has been suggested that "for many librarians, assessment is limited to rote counting of anything and everything that is amenable to counting" (Sandler, 1992, p.184). Research in librarianship has followed the general trend in the social sciences to place value on the quantitative paradigm of the natural sciences. Recently, however, the belief of the qualitative researcher that human behavior cannot be measured in this way has received greater respect.

  5. The dichotomy of quantitative and qualitative research has arisen because the manner in which knowledge about the world is acquired has been understood within one of two paradigms. The dominant natural science paradigm (Patton, 1991) has traditionally guided researchers in the social sciences. This approach necessitates that the researcher believes the positivist view that fundamental reality lies in physical objects and processes. Thus, human behavior is operating according to cognitive rules or as a result of mechanical forces. The positivist believes that knowledge can be guided by observable facts and that human behavior is sufficiently uniform to be able to make accurate forecasts on the basis of statistically representative samples. Thus, universal statements of truth can be made from quantitative data collected about cause and effect relationships. Data is subjected to statistical analysis in order to test pre-conceived hypotheses.

  6. Qualitative research has developed as researchers have found that "[t]his use of quantitative analysis and mathematical formulae in the natural sciences is not possible in the social sciences because the concepts found in social theories can only be measured qualitatively" (Blaikie, 1993, p. 19).

  7. The qualitative researcher employs the phenomenological approach that naturally occurring activity is worthy of investigation. This can be distinguished from the positivist belief in the importance of certainty, which suggests that investigation should be carefully structured and conducted under controlled conditions. Qualitative research is also concerned with the primacy of the mind as the fundamental stimulus for human conduct. This means that primary data is generated by the person under investigation, rather than empirical data derived from observable processes. Patton (1991) describes this approach as the "representational theory of the mind" (p. 391). Knowledge is a representation of the world which has been cognitively worked by the mind, and physical reality is, therefore, only known by an act of mind on phenomena presented to it.

  8. Qualitative investigation is concerned not with cause and effect but with understanding, so inquiry is a never-ending process as multiple interpretations are possible, based on individuals and contexts. Instead of testing hypotheses, theories emerge as the data is recorded. Advocates of grounded theory maintain that pre-conceived methods of data collection ignore the contextual relevance of behavior and therefore remove its relevance. Instead, grounding the theory in the data is advocated (Denzin & Lincoln, 1994, p. 204). This process, known as grounded theory, is described by Weingand (1993) as "[i]nstead of beginning with a hypothesis that must be proven either true or false, casting a broad net for data allows the theory to emerge over the process of data collection--grounding the developing theory in the data itself" (p. 20).

  9. It is not necessary to adhere to one of the above paradigms to the exclusion of the other. Methodological pluralism, or diversity, has the advantage of providing data which is complementary, strengthening the likelihood of valid and reliable results (Wildemuth, 1993; Patton, 1991). The important concern is to ensure that methods are employed flexibly so that problem solving is effective (Weingand, 1993). However, logically incompatible findings will result from mixing paradigms without conscious awareness that one is doing so (Patton, 1991). It is also possible to use more than one method of data collection, a technique known as methodological triangulation, in order to overcome any deficiencies in method (Fidel, 1993). It is important to maintain an awareness of which approaches are being used to ensure that results are compatible.

  10. The decision about qualitative or quantitative research techniques depends upon the ontological and epistemological assumptions of the researcher and goals of the study. In other words, we should ask what it is we are seeking to know and how such knowledge might best be acquired. The question posed will determine which method or methods are appropriate for collection, and these methods will then govern the nature of the data itself.


    METHODOLOGICAL APPROACHES IN END-USER STUDIES

  11. Ephraim (1994) believes that the reliance on quantitative methods in librarianship arose from its popularity in the social sciences, with which library science wished to be associated (p. 237). We are warned that statistics are susceptible to misleading presentation and interpretation and a carefully made judgement can be just as reliable. Sturges (1995) suggests a need to move beyond enumeration towards understanding.

  12. Recent investigations into end-user searching have made use of the grounded theory technique of qualitative research (Ellis, 1993; Wildemuth, 1993). Data is descriptive, and the process is iterative, with tentative hypotheses being continually checked against the data and revised accordingly (Bradley, 1993; Heyink & Tymstra, 1993). Thus, components are derived from the data itself, rather than trying to force the data into preconceived, rigid structures in order to make it quantifiable (Weingand, 1993, p. 19). Samples used in qualitative research can be distinguished from the probabilistic sampling used in quantitative research. A small sample is chosen, based on appropriateness to the topic. Bradley (1993) suggests that selection is made of samples that are extreme, information-rich, critical or typical cases. The qualitative approach can be seen as appropriate for an end-user study as these methods "tend to give more attention to the subjective aspects of human experience and behavior" (Glazier & Powell, 1992, p. xi).

  13. Ellis (1993) advocates searching for categories and the consistent relation of incidents to those categories, allowing their refinement and integration until several core categories remain, a process he describes as "analytical exhaustivity" (p. 478). However, the researcher should approach qualitative research with caution. Ellis argues that this process of deriving categories and properties and demonstrating their relationship is "the most creative and intellectually demanding part of the researcher's task" (p. 479).

  14. Wildemuth (1993) used "methodological pluralism" in a study aimed at discovering the relationship between online searching and domain knowledge. The number of moves made and their frequency was calculated, and protocols were used to test the hypothesis that regular use of the 'repeat' key was related to accidental exits from the display. Wildemuth concluded that it is possible to combine the generalisability of quantitative research and the detail of qualitative techniques so that "the two approaches richly inform each other" (p. 466).

  15. Hancock-Beaulieu, Fieldhouse, and Do (1995), in an investigation into interactive query expansion, use qualitative data derived from questionnaires to complement the quantitative data collected from transaction logs. Su (1991) suggests that quantitative data relating to the length of a search and number of documents retrieved can be usefully combined with qualitative data relating to the satisfaction of the user with the search.

  16. Fidel (1993) argues that librarianship, although a social science, should "develop a methodological framework and a set of procedures adequate for its own research" (p. 234).


    DATA COLLECTION

  17. Whether the social science researcher chooses a quantitative or qualitative methodology, there are three basic techniques available to collect the data: 1) interviews (both face to face and via questionnaires); 2) observation; and 3) record examination.

  18. It is important that the technique or techniques chosen are carefully considered to ensure maximum validity and reliability (Mason, 1994, p. 104; Altheide & Johnson, 1994). The validity of a technique is its ability to provide an accurate measure of a concept which is relevant to the hypothesis being tested. For it to be reliable, it must be consistent so that any change should be the result of a change in the characteristic being measured, not the result of an unreliable measuring instrument. This is also referred to as "test-re-test reliability" (Bailey 1987, p. 72). This is particularly important if the research is expected to be large-scale, involving more than one researcher.

    Interviews/Questionnaires

  19. An analysis of previous research in end-user searching has shown that questionnaires as a method of data collection have proven consistently popular. The questionnaire offers subjective data, which can be used to assess what people perceive to be the case. This can be a useful approach if the perception of the user is an integral part of the research. Ankeny (1991), for example, undertook research to ascertain whether user satisfaction equaled success. To this end, the questionnaire proved a useful tool, especially as a sample of 600 users prohibited the use of interviews. Hancock-Beaulieu et al. (1995) used the questionnaire to gain qualitative data to complement quantitative data from transaction logs, an approach which enabled research to evaluate system/user interaction (p. 235). Hsieh-Yee (1990) also combined quantitative and qualitative methods in order to be sure that the most reliable data was gained, but argued that while questionnaires and interviews are an effective way of gathering information about user satisfaction and impressions, these methods will not allow any measurement of system interaction. Su (1991) was interested in designing a model to help improve system development, and found that the questionnaire could play a part in a quantitative/qualitative study. Ensor (1992) agrees that the questionnaire is not necessarily accurate, but is the only way short of interviews to elicit characteristics and subjective opinions. Interviews may be unfeasible due to a large sample size.

  20. If at all possible, interviews should be carried out, as they offer a number of advantages over the questionnaire. They are flexible, ideas can be followed up, and reactions probed further. The response rate tends to be higher and more complexity is possible. (Bailey, 1987). According to Sandler (1992), "open-ended questions by trained and neutral interviewers can determine what the interviewee considers important" (p. 182). Denzin and Lincoln (1994) argue that "the interview is the favourite methodological tool of the qualitative researcher" (p. 353).

  21. Dyer and Bouchet (1995) used interviews to discover users' opinions about the relevance of references received. Questionnaires were also employed, in order to gain basic demographic data and measure how users evaluated the information that was retrieved. Su (1991) made use of the interview to try and eliminate some of the problems encountered in questionnaire-based research, such as ambiguous wording, ill-defined categories and inadequate measuring instruments.

    Observation

  22. According to Ellis (1993, p. 471), the ideal or root method of research is observation. Bailey (1987) argues that it is the primary technique for the collection of non-verbal data and Bell (1993) suggests that this method reveals characteristics which are impossible to discover by other means. Certainly, if the area of interest is what people do, rather than what they say they do, or if there are reasons to believe that the one will differ from the other, then observation is a research method worthy of careful consideration. Bailey (1987) is of the opinion that,

    [i]f a researcher is interested in gathering data on human actions, as oppose to beliefs, values, or opinions, direct observation of the act by the researcher would seem to have superior face validity over data collected by questionnaire . . . and document study. (p. 265)
  23. However, there are drawbacks to observational studies regarding reliability. Quantification of data can be complex (Sandler, 1992, p. 180), and it is debatable how consistently two researchers will observe the same phenomena. Adler and Adler (1994) maintain that validity is brought into question as researchers have to rely on their own subjective interpretation. Also, it can be argued that a subject aware that he or she is under observation will act differently (Mullings, 1984). They might feel embarrassment or the urge to 'show-off,' both of which would have adverse effects on the reliability of the data. A potential solution would be to observe anonymously, but this might be difficult if the observation study was lengthy or complex. Conversely, Anderson (1995) argues "that most of the subjects did not find [the presence of the researcher] of particular interest once they were aware that [the] sole purpose was to gather data" (p. 367).

  24. Mullings (1984) comments that observation has been an under-used technique in librarianship, but argues that "observation could play a useful part in library research, especially if used in conjunction with other methods" (pp. 7-8).

  25. At the same time, it is acknowledged that care is needed when deciding how to approach this method of investigation. Decisions have to be made about whether the study is to be overt or covert, structured or unstructured, direct or indirect, in order to be satisfied that data yielded will be useful.

  26. Observation will allow the researcher to record what is actually happening, but it will not necessarily reveal why it happens. For this reason many researchers have encouraged subjects to think aloud, which can be used to assist in the recording of quantitative data or transcribed in full and subjected to a qualitative analysis. Dalrymple and Zweizig (1992) used transcripts of such "think aloud protocols" to measure reformulation behavior. This data was combined with questionnaire results which were analyzed quantitatively. This is a user- rather than a system-based analysis which, they suggest, is better suited to research on user behavior (pp. 177-178). Hsieh-Yee (1990) suggests that system- and user-based research be combined, using protocols in association with transaction logs. A potential problem with protocols is that they could endanger the validity of the data. Su (1991) suggests that video or audio recording of protocols may affect the actions of some users, and dissuade others from taking part entirely (p. 134).

    Record Analysis

  27. This is a method concerned with the examination of data in written form. For the purposes of a study of information retrieval, this would involve an examination of the actual data retrieved from the system. It is approached in a manner similar to an observation study, determining variables and establishing a method for measuring them. Again, it is vitally important that care is exercised in order to ensure that data is reliable.

  28. Record analysis is a social research method which is available to, and perhaps vital to, the researcher into end-user studies. Conclusions can be drawn from an analysis of the search strategy or the actual information retrieved and many systems make it possible to record searches over a period of time (transaction logs). Researchers into end-user searching on library OPACs have used this method (Ballard, 1994). Bysouth (1990) found logs useful to record the searches of research scientists, but recognized that "we needed to acquire not only the details of actual queries executed and strategies employed but also the user response to the various selected approaches together with their level of satisfaction and the results obtained" (p. 111).

  29. According to Sandore (1993), the transaction log offers the advantage of allowing unobtrusive observation. Hsieh-Yee (1990), however, argues that transaction logs, while unobtrusive and facilitating ease of data collection, are limited in advancing our understanding of thought processes, and Mitev, Venner, and Walker (1985) agree that "there is much that cannot be deduced from even the most detailed transaction logs" (p. 176).

  30. Research has also used transaction logs to evaluate searching of full-text databases (Nicholas, 1995), but it can be argued that this method lends itself more to simple searches which are designed to retrieve a particular piece of information, rather than the measurement of serendipity and iterative techniques which are employed in the use of full-text databases. For example, Wildemuth, Jacobs, and Fullington (1991) employed this method during research into the use of a database of facts and concepts in microbiology: "To interpret the results, it is important to remember that searchers were trying to identify one particular record in the database that would satisfy each assigned problem" (p. 310).

  31. Ford, Wood, and Walsh (1994) employed content analysis in a qualitative manner, by asking the student to judge the relevance of the documents retrieved. This was complemented by the well-established quantitative techniques of precision and recall.


    COMPARATIVE STUDIES

  32. Finally, any or all of the above methods can be incorporated into a comparative study, where the aim is to compare one phenomenon with another. In the case of end-user searching, it is obviously appropriate to consider how the searching behavior and results of the student compare to those of a professional librarian. Several studies have made use of this technique (Ballard, 1994; Hsieh-Yee, 1990; Marchionini, Dwiggins, & Katz, 1993).


    CONCLUSION

  33. The above study of the literature suggests that a combination of methods in order to maximize the results is beneficial. The final decision on a methodological framework should be based on three things:

    1. The epistemological and ontological beliefs of the researcher. Methods employed should be in harmony with the researcher's own views about the sources of knowledge and the manner in which this knowledge may be discovered.
    2. The epistemological and ontological goals of the study. The relative merits of positivist or phenomenological approaches are dependent on the type of data required and the subject being studied. Reasons for human behavior might best be determined by qualitative methods, whereas a comparison of the performance of two computer systems using predetermined variables could be carried out quantitatively.
    3. The size of the study. Even the most committed of qualitative researchers must admit that very large studies do not lend themselves to such an approach. The amount of data generated can be vast and the analysis time-consuming. The researcher should be prepared to be flexible and adopt a combination of methods that provide the best compromise when faced with a conflict between desired research methods and practical necessities.

  34. The final decision must rest with the researcher who needs to consider the advantages and disadvantages of the different approaches, bearing in mind the possible benefits of methodological pluralism. The need to ensure the success of the particular piece of research should be regarded as of prime importance, and not the sociological and philosophical arguments about methodological paradigms.


REFERENCES

Adler, P. A., & Adler, P. (1994). Observational techniques. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research. Thousand Oaks, CA: Sage Publications.

Altheide, D. L., & Johnson, J. M. (1994). Criteria for assessing interpretive validity in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 485-499).Thousand Oaks, CA: Sage Publications.

Anderson, J. (1995). Have users changed their style?: A survey of CD-ROM vs. OPAC product usage. RQ, 34(3), 362-368.

Ankeny, M. L. (1991). Evaluating end-user services. Journal of Academic Librarianship, 16(6), 352-356.

Bailey, K. D. (1987). Methods of social research (3rd ed.). New York: The Free Press.

Ballard, T. (1994). Comparative searching styles of patrons and staff. Library Resources and Technical Services, 38(3), 293-305.

Bell, J. (1993). Doing your research project: A guide for first-time researchers in education and social science (2nd ed.). Philadelphia, PA: Open University Press.

Blaikie, N. (1993). Approaches to social enquiry. Cambridge, UK: Polity Press.

Bradley, J. (1993). Methodological issues and practices in qualitative research. Library Quarterly, 63(4), 431-449.

Bysouth, P. T. (1990). Evaluating the use of several approaches to online literature retrieval by research scientists. In P. T. Bysouth (Ed.), End-user searching: The effective gateway to published information (pp. 105-123). London: ASLIB.

Conkling, T. W., & Osif, B. A. (1994). CD-ROM and changing research patterns. Online, 18(3), 71-74.

Corral, S. (1995). Academic libraries in the information society. New Library World, 96(1120), 35-42.

Dalrymple, P. W., & Zweizig, D. L. (1992). Users' experience of information retrieval systems: An exploration of the relationship between search experience and effective measures. Library and Information Science Research, 14(2), 167-181.

Denzin, N. K., & Lincoln, Y. S., (Eds.). (1994). Handbook of qualitative research. Thousand Oaks, CA: Sage Publications.

Dyer, H., & Bouchet, M-L. (1995). A comparison between the perceived value of information retrieved via end-user searching of CD-ROMs and mediated online searching (BLRD Report 6208). British Library.

Ellis, D. (1993). Modeling the information seeking patterns of academic researchers: A grounded theory approach. Library Quarterly, 63(4), 470-484.

Ensor, P. (1992). User practices in keyword and boolean searching on an online public access catalog. Information Technology and Libraries, 11(3), 210-219.

Ephraim, P. E. (1994). A review of qualitative and quantitative measures in collection analysis. The Electronic Library, 12(4), 237-241.

Fidel, R. (1993). Qualitative methods in information retrieval research. Library and Information Science Research, 15(3), 219-247.

Ford, N.; Wood, F.; & Walsh, C. (1994). Cognitive styles and searching. Online and CDROM Review, 18(2), 79-86.

Glazier, J. D., & Powell, R. R. (1992). Qualitative research in information management. Englewood, CO: Libraries Unlimited.

Hancock-Beaulieu, M.; Fieldhouse, M.; & Do, T. (1995). An evaluation of interactive query expansion in an online library catalogue with a graphical user interface. Journal of Documentation, 51(3), 225-243.

Heyink, J. W., & Tymstra, T. (1993). The function of qualitative research. Social Indicators Research, 29(3), 291-305.

Hsieh-Yee, I. P. Y. (1990). The search tactics of novice and experienced searchers: A comparative study. University of Wisconsin-Madison.

Joint Funding Council's Libraries Review Group. (1993). Report [Follett Report]. (Chairman: Professor Sir Brian Follett), The Councils.

Marchionini, G.; Dwiggins, S. S.; & Katz, A. (1993). Information seeking in full-text end-user oriented search systems: The roles of domain and search expertise. Library and Information Science Research, 15(1), 35-69.

Mason, J. (1994). Linking qualitative and quantitative data analysis. In A. Bryman & R. G. Burgess (Eds.), Analyzing qualitative data (pp. 89-110). London: Routledge.

Mitev, N. N.; & Venner, G. M.; & Walker, S. (1985). Designing an online public access catalogue: Okapi, a catalogue on a local area network (Library and Information Research Report 39). The British Library.

Mullings, C. (1984). Observation. Centre for Research in User Studies, University of Sheffield.

Nicholas, D. (1995). Are information professionals really better online searchers than end-users? In 19th National Online Information Meeting Proceedings (pp. 383-397). Learned Information Europe.

Patton, M. J. (1991). Qualitative research on college students: Philosophical and methodological comparisons with the quantitative approach. Journal of College Student Development, 32(5), 389-396.

Sandler, M. (1992). Qualitative research methods in library decision making. In J. D. Glazier & R. R. Powell (Eds.), Qualitative research in information management. Englewood, CO: Libraries Unlimited.

Sandore, B. (1993). Applying the results of transaction log analysis. Library Hi Tech, 11(2), 87-97.

Sturges, P. (1995). Field research in information studies: A Malawian study. In Proceedings of the 1st British-Nordic LIS Conference, Copenhagen, 22-24 May, 1995. Royal School of Librarianship, Denmark. Available from < http://epip.lut.ac.uk:80/bailer/35-sudrg.htm>

Su, L. T. (1991). An investigation to find appropriate measures for evaluating interactive information retrieval. Unpublished doctoral dissertation. Rutgers, State University of New Jersey.

Tenopir, C., & Neufang, R. (1995). Electronic reference options: Tracking the changes. Online, 19(4), 67-73.

Weingand, D. E. (1993). Grounded theory and qualitative methodology. IFLA Journal, 19(1), 17-26.

Wildemuth, B. M. (1993). Post-positivist research: Two examples of methodological pluralism. Library Quarterly, 63(4), 450-468.

Wildemuth, B. M.; Jacobs, E. K.; & Fullington, A. (1991). A detailed analysis of end-user search behaviors. In Proceedings of the ASIS Annual Meeting, 28, 302-312.