Appraisal, Provenance, and the Computer Revolution:
An Examination of Organizational
Records in the Electronic Age

Harold E. Thiele, Jr.
Department of Library and Information Science
University of Pittsburgh


© 1998 Harold E. Thiele, Jr. hthiele@lis.pitt.edu, Department of Library and Information Science, University of Pittsburgh
Katharine Sharp Review ISSN 1083-5261, No. 6, Winter 1998 [http://edfu.lis.uiuc.edu/review/6/thiele.html]

    One of the by-products of the Second World War was the computer. Originally conceived as an electro-mechanical device capable of performing the repetitive numerical operations needed to produce ballistics tables, the computer slowly evolved into the multi-functional machine that dominates offices throughout the world. The development of the multi-functional computer and the introduction of high speed networking technology is changing how individuals and businesses perform their daily operations, and, as a consequence, the type of records they are producing. As the tsunami of technology surges onward, efforts are being made to understand the changes and consequences of the computer revolution. These changes are calling into question many of the traditional assumptions archivists and records managers use to appraise and classify records. This article examines some of these assumptions and explores proposed solutions to the problems relating to appraisal and classification of organizational records in the electronic era.

    Whenever I'm confronted with a choice between two evils, I kinda like to go for the one I never tried before.
    --Mae West in the film My Little Chickadee (cited by Burke, 1988, p. 47)

    INTRODUCTION

  1. The basic assumptions and practices followed by archivists and records managers in handling organizational records have their beginnings in Europe several hundred years ago. With the rise of the European nation-states over the last 400 years, authority was transferred from localized feudal bodies to more centralized administrations. This led to the rise of civil administrators and the bureaucracy. One side effect of this was the development of written regulations and documentation of decisions made based on these regulations (Lutzker, 1982).

  2. These basic assumptions grew out of an environment bounded by a print-based (i.e., parchment and paper) universe. Print-based materials, at this time, were very expensive and not to be wasted. In each and every case, the 'record' of the transaction became indistinguishable from the print-based medium that contained it. Unbound documentary materials developed into transportable forms that identified the type of transactions they were documenting. Invoices, deeds, bills of lading, and passports are examples of these portable documentary types. Often documentary materials were bound into books or journals dedicated to documenting a single type of transaction. Examples are accounts receivable ledgers, day journals, and books of vital statistics. These records of business transactions tended to remain physically in the office that created them. This was also an era when it was assumed that an office would perform the same function year in and year out. The function, the office and the records were inseparable.

  3. At the beginning of the Twentieth Century, Weber's analysis of the bureaucracy, with its ideal of institutional hierarchy and administrator roles in decision making, was incorporated into the archivist's consciousness (Lutzker, 1982). To the assumption that the function, office, and record were indistinguishable was added the concept of a stable organizational structure. During this period, the most efficient method of maintaining control over records was to keep each office's records together based on organizational structure. Even if the records were relocated to an archive or storage facility, they were kept together and identified by their submitting office, and not by their function or activity. There was little difficulty in locating records dealing with a particular business transaction or activity since they tended to be print-based documents localized in one identifiable office.

  4. The advent of the electronic office has called into question many of these assumptions. Since World War II, records associated with particular business transactions or activities are no longer confined to an individual office within a single organization. Many of these transactions involve multiple political entities, corporate bodies, and individuals. Some parts of the documentary evidence are captured on print-based medium, while other parts are captured by electronic or other forms of media. Within the distributed electronic office, the documentary evidence for the business transaction may not reside within the office, the state, or even the nation state. Parts of the documentary evidence may reside on many different computers in many different locales. In addition, the electronic databases do not conform to the traditional definitions of various record types. Where the information was maintained in separated record formats in the print-based world, in the electronic world the information is merged into a single relational database, and the information is produced according to specified reporting formats. Finally, unlike the print-based records, one cannot define an electronic record just by looking at it. Electronic records require interpretation by some additional software package before it can be understood.


    PROBLEM

  5. The increasing size and complexity of modern civilization is generating increasing quantities of records. We are being overwhelmed by the records we create. Without appraisal and its associated records destruction we would be buried in outdated records. Society records more information than it needs. Much of it is of an ephemeral nature or highly repetitive and not necessary to maintain (Boles & Young, 1991). By 1990 the United States was creating over 1.3 trillion documents each year (Toffler, 1990). The pace of record keeping is increasing. Roy Williams of the California Institute of Technology estimates that the amount of printed information is currently equivalent to 200 petabytes (a petabyte is a quadrillion bytes or 10E15 bytes). Williams also estimates that the amount of information stored electronically is expected to exceed 500 petabytes by the year 2000 (Murphy, 1996).

  6. The sheer mass of records defeats any attempt to obtain an intellectual understanding of the events being documented. There are over five billion documents stored at the National Archives in Washington, D.C. There are an additional 19 million cubic feet of federal documents stored in federal repositories located across the United States (Murphy, 1996). In addition to these stored federal records, there are several times this amount stored in thousands of other governmental (state and local), corporate, institutional, and private repositories located around the country. It is estimated that only 1% to 5% of the total records created are worthy of preservation as records of continuing value (Reed, 1993, p. 158).

  7. The appraisal process reduces redundancy and thereby "make[s] accessible and interpretable, for archival and research purposes, the intellectual working tools of organizations. [The] . . . aim of archival appraisal . . . should be to make archives eloquent and to facilitate research" (Menne-Haritz, 1994, p. 530). Records are created as "intellectual working tools for the steering and controlling of cooperative decision-making processes" (p. 541). It is only by removing the outdated, redundant, and generally valueless records that this process becomes clear.


    APPRAISAL

  8. The traditional definition of appraisal is the process of evaluating documents to determine their value and disposition (National Archives and Records Administration, 1993; Robek, Brown, & Maedke, 1987). Cox and Samuels (1988) use a much broader definition, "any selection activity that enables archivists to identify recorded information that has enduring value, primarily for the documentation of society" (p. 29).

    The basic concept that underlies appraisal practice is record value. The concept hypothesizes that certain values inhere in records, that these values are primarily defined by use, and that the archivist should be able to identify and measure these values. (Ham, 1993, p. 7)

  9. Documents are evaluated according to a set of criteria and classified as to how long they are to be retained, where they are to be maintained, who shall have access to them, the conditions of the access, and the manner of their disposal. Retentive value depends on "[the] continuing importance of records, taking into account their administrative, legal, and fiscal use, their research value, and their relationship to other records" (Dearstyne, 1993, pp. 1-2).

  10. The critical aspect of the appraisal process to keep in mind is its decisiveness. Because of the uniqueness of most archival materials, the archivist's decision about what records are to be maintained and which are to be destroyed is irrevocable (Ham, 1993). The responsibility of the archivists to select documents and records for destruction often upsets many people because of the finality of the decision. However, it is only through the appraisal process that we are able to exercise some control over the records we create.

  11. Appraisal is an iterative process that seeks to separate out records that have long term or continuing value from the great mass of records that have only ephemeral value. This process initially separates a collection of documentary materials into two distinct classes: records and nonrecords. It is with this primary classification that the impact of the computer is first felt.


    RECORD

  12. A key issue that must be dealt with is the elusive concept of a record. Everyone 'knows' what a record is, but no one seems to be able to provide a definition that satisfies everyone. The first step in the appraisal process is identifying what is and is not a record. The better definitions attempt to provide some form of functional definition. This said, the overall impression is that the focus is still on record formats fixed on print-based media.

  13. Records management textbooks tend to define a record as any recorded information that a business produces or receives, regardless of its characteristics, medium, or form (Robek, Brown, & Maedke, 1987). The Society of American Archivists prefers a more restricted definition: "a document created or received and maintained . . . in pursuance of legal obligations or in the transaction of business" (Bellardo & Bellardo, 1992, p. 28).

  14. Most laws in North America take a similar approach to defining a record, but usually include an extensive listing of various records types and add a long list of exemptions to the law. The model example of this approach is from the Public Printing and Documents Act (1991) which defines records as:

    [including] all books, papers, maps, photographs, machine-readable materials, or other documentary materials, regardless of physical form or characteristics, made or received by an agency of the United States Government under Federal law or in connection with the transaction of public business and preserved or appropriated for preservation by that agency or its legitimate successor as evidence of the organization, functions, policies, decisions, procedures, operations, or other activities of the Government or because of the informational value of data in them.

    Canada also employs a similar style of language in defining what is meant by a record but exclude the 'evidence' section that is present in the United States Code (Canada, 1991; Canada, 1992).

  15. In the legal codes of the various states the term used is often 'public records.' A good example of this approach can be found in the North Carolina statutes:

    'Public record' or 'public records' shall mean all documents, papers, letters, maps, books, photographs, films, sound recordings, magnetic or other tapes, electronic data-processing records, artifacts, or other documentary material, regardless of physical form or characteristics, made or received pursuant to law or ordinance in connection with the transaction of public business by any agency of North Carolina government or its subdivisions. (North Carolina, 1995, p. 132-1)

    In this section, 'public records' are defined in a manner similar to that used to define 'records' in the United States Code. As with the Federal statute, the emphasis is placed on the record format (i.e., letters or maps) or on the media used to record the record (i.e., film or tapes). As one delves into the laws governing records, fine distinctions are uncovered. In Minnesota, distinctions are made between 'government records,' 'state records,' and 'local records.' In addition, 'records' are defined to exclude data and information not considered to be part of an official transaction (Minnesota, 1994, Chapter 138.17 Government Records; Administration, Subdivision 1(b)).

  16. Record definitions are not limited to statutes. Governmental agencies and departments issue rules and regulations governing records. As in the previous case, these definitions mirror the Federal code approach of listing record formats or media. The EPA (1994) definition is an example of this approach at the departmental level:

    Official records are made or received in the conduct of Agency business. They relate directly to the functions of an office and they document those functions. They consist of recorded information of continuing administrative, fiscal, legal, historical, or informational value, including published materials, papers, maps, photographs, microfilm, audiovisual, optical disks, electronic records (computer tapes, disks, floppy diskettes) or other documentary materials regardless of physical form or characteristics, made or received by the Agency that evidences the organization, functions, policies, decisions, procedures, operations or other activities of the Government. (Attachment A)

  17. In addition, there is the concept of 'nonrecord materials' which refers to materials excluded from or not meeting the legal definition of records (National Archives and Records Administration, 1993). This usually refers to documentary materials of a personal nature, that are not related to official business. Often there are extensive lists of materials exempted from the definition of public record. For example, Title 1 V.S.A. 317 (Vermont, 1995) provides a listing of 22 exemption categories or classes that include, among other items, library materials and test questions.

  18. The increasing use of computers to produce and maintain records is resulting in a re-examination of the records concept. There is a growing awareness that the traditional definitions of records, with their associated lists of media dependent record types, are becoming outdated and unworkable. For example, references to account books or maps have no meaning in an electronic world. In an electronic system, the financial information exists in a complex relational database, from which an account book may be generated. A map doesn't exist in a Geographic Information System (GIS) either. It is just a collection of data points and vector references. This disconnection of the information from a specific media presentation is driving a movement toward developing a functional definition of the record that will exist independent of any media.

  19. David Bearman (1994), one of the leading proponents of a transaction based approach that seeks to have a dynamic records definition, states that,

    [a] record is any communication between one person, and another, between a person and a store of information available to others, back from the store of information to a person or between two computers programmed to exchange data in the course of business. The important aspect of this definition is that a record is not a collection of data but the consequence of a business event. Records 'occur' rather than 'are.' (p. 190)

    This approach incorporates the concept that the transaction record provides evidence that the activity has occurred and the manner in which it has occurred. As Roberts (1994) states, "transactional records are recorded information [taken] . . . in the transaction of business or the conduct of affairs and kept as evidence of such activity" (p. 17). He further reports that at the Records Management Office of New South Wales, Australia they distinguish between electronic documents and electronic records "by the transactional origins and evidential qualities of the latter" (p. 21).

  20. Australia has made the most progress in adapting this concept of record into law. In 1995, Standards Australia issued a draft standard for records management that included functional definitions of records and record keeping. Record keeping is defined as "the making and maintaining of complete, accurate and reliable evidence of business transactions in the form of recorded information" (pp. 4, 7); transactional records are defined as "recorded information, in any form, including data in computer systems, created or received and maintained by an organization or person in the transaction of business or the conduct of affairs and kept as evidence of such an activity" (p. 8); and a transaction is defined as "the smallest unit of business activity" (p. 8). This standard was approved in 1996 for use in Australia. It has been submitted to both the International Standards Organization (ISO) and the North Atlantic Treaty Organization (NATO) as a proposed international standard. At this time it is currently being reviewed by both agencies for adoption.

  21. Dollar (1992) clarifies the changes that the electronic revolution is making in the record keeping world. He points out that many of the traditional records definitions depend on the secondary characteristics associated with paper records to define the record--characteristics that are no longer available in the electronic world. A transactional definition that captures the evidence of the activity taking place will ensure the future of record keeping activities. In the new electronic world, the record is no longer tied to its container.


    PROVENANCE

  22. Since records result from organized and purposeful activity, the Principle of Provenance asserts that important information can be obtained from the context and relationships surrounding the records. The Principle of Provenance developed out of administrative rules in Denmark (1791), Naples (1812), the Netherlands (1826), and France (1841). The first published theoretical expression of the principle was by Francesco Bonaini in 1867, who "articulated the principle of maintaining all the documents of a body in their original order" (Eastwood, 1992, fn 4, p. 13).

  23. The original French concept of respect des fonds had both external and internal dimensions. The external dimension stressed identification of records with their office of creation, and the internal dimension stressed maintaining the original record order. The Germans articulated in Provenienzprinzip a greater precision in maintaining the integrity of records with the record creator. This German articulation of the respect des fonds became known as provenance (Cook, 1992b).

  24. The Principle of Provenance emphasis is on the origin of the records. It seeks " . . . the preservation of the context of the records, that is their links to purpose, function and activity, to the individual or parts of an organization which created them, and to other records created by that individual or within that organization" (McKemmish, 1993, p. 10).

  25. The context of records creation is more important than the contents of the records themselves (Dearstyne, 1993; Reed, 1993). As Dollar (1992) succinctly puts it, "understanding the provenance of records requires an understanding of the business functions and activities underlying the information system that produced the records" (p. 49). Bearman and Lytle (1985) maintain that the effectiveness of provenance "rests on a detailed understanding of both the structure and processes of the organizations which created the records in question" (p. 16).


    ORIGINAL ORDER

  26. In addition to the Principle of Provenance, archivists and records managers have traditionally used the Principle of Original Order as part of the appraisal process. The Principle of Original Order, the internal dimension of respect des fonds, maintains that records should be kept in the order in which they were organized or maintained by the creating agent. In this fashion the original order "documents the sequential development of organic activity, reveals organizational approaches, shows administrative processes, and in general documents relationships" (Dearstyne, 1993, p. 130). It provides evidence of the manner in which a particular activity or transaction was performed, the particular order in which a function was fulfilled (Brunton & Robinson, 1993).


    PRINT- vs. ELECTRONIC-BASED ENVIRONMENTS

  27. The key impact that the electronic environment is having on appraisal is the dislocation of the 'record' from its container. With print-based records it is comparatively easy to make use of the Principles of Provenance and Original Order to appraise the records because

    provenance related information in paper files is manifested in organization charts and manuals, procedures, policies, physical arrangement of the records, media characteristics such as watermarks, formal elements such as letterheads, and in content/context elements such as names of writers and recipients, dates, references to other documents or files, and the like. In addition, individual documents may contain marginalia, initials, and similar contextual information that reveal the information environment in which records were created, used, and maintained. Because the physical, formal, and contextual components of traditional records are inseparable, the physical location of one record vis-a-vis another record denotes how records were created and used at a given point in time. (Dollar, 1992, p. 49)

  28. Electronic records, on the other hand, provide no physical expression that can be used to determine their provenance. The most glaring difference between electronic and print-based records is that an electronic record cannot be viewed without the aid of some intervening software and hardware. Without the correct software and hardware, the electronic data cannot be seen to determine whether or not it is a record.

  29. The second area of concern is that an electronic record is stored as a randomized pattern of bits throughout the storage area, intermixed with randomized bits from other records and nonrecords. The provenance or the original order of these bits cannot be determined. Computer memory management systems compress data that is not in active use and or restructures the data in some other fashion to maximize the use of storage space. In networked environments, it is often difficult to identify which linked computer system has which part of the record.

  30. A third area of concern is the current process of merging information from many offices into a centralized data warehouse. Since each bit of information may or may not be used by a wide variety of offices, there is great difficulty in identifying which office has custody over which bit of data. Custody is an important aspect of appraisal because it often determines the retention period for maintaining records and their associated data.

  31. A fourth area of concern is replacement of earlier data with newer data. In many data warehouses, no record is maintained of old data and their links to previous activity. Geographical Information Systems (GIS) are a good example of this. As new data comes in, the old data is often overwritten or deleted from the system. It is often impossible to recreate the status of the system as it existed months or even weeks earlier. Without being able to reconfigure the system to an earlier status, it is impossible to reproduce the information and its presentation that project managers used to made decisions. This means that there is no source record available for the decisions that affect entire populations, regions, and ecosystems.

  32. A fifth area of concern is the question of whether the preservation of the electronic record is long enough so that it can be appraised. Unlike print-based records, electronic records require early and active intervention to ensure their preservation and maintenance. If these records are not upgraded whenever the system is changed, they have a good chance of becoming unreadable, and therefore unappraisable. If these records are not transferred to fresh media, the media that they are recorded on will probably degrade and become unreadable.

  33. A sixth area of concern is the ease of duplication and lack of authentication associated with many computer systems. Individuals are able to get copies of as many records as they want. Changes can be made very easily to electronic records, and in most systems are undetectable and/or untraceable. It usually is impossible to tell which copy is the "original" and which is the duplicate.


    TRADITIONAL APPRAISAL CATEGORIES

  34. Until the Twentieth Century there was little need for appraisal methodology. With the development of the modern nation state and its attendant bureaucracies, the mass of records threaten to overwhelm resources. While work on appraisal methodology had begun in Europe before the beginning of the Twentieth Century, it wasn't until the 1940s that systematic work began in the United States. The pioneering work of Philip C. Brooks, G. Philip Bauer, and Theodore Schellenberg defined the basic criteria that were to guide archivists around the world in appraising records for the next forty years (Boles & Young, 1991).

  35. In the traditional appraisal process, two characteristics are considered to be of importance. The primary characteristic is the evidential or documentary value of the record. The secondary characteristic is the informational value of the record.

  36. The evidential value of the record is reflected in the evidence it provides on how an office or individual conducted its business. The three major types in this category are records that: 1) have continuing administrative, legal, or financial use; 2) protect individual or community civic, legal, property, or other rights; or 3) document the development or evolution of the creating body or individual (Reed, 1993).

  37. The informational value of the record is determined by its information content and the probability that it will be of use to the originating organization or to a variety of researchers (Reed, 1993).

  38. This traditional methodology depends on a bottoms-up approach to the examination of the records and their classification for retention or destruction. The records come to the attention of the archivist near the end of the their life cycle, when their usefulness to the creating organization is greatly diminished. The most widespread complaint about this approach is that the records are appraised separately from their social and organizational context (Reed, 1993).

  39. In the 1970s Brichford expanded on this traditional model and added additional criteria and categories, including an archival category. This category looked at the relationship of the records in question to other records and the costs associated with processing, preservation and storage (Boles & Young, 1991).

  40. In 1985, Haas, Samuels, and Simmons challenged the archival world to examine the functional process and communicational patterns associated with their institutional enterprises.

    To appraise effectively, archivists need to understand that the nature of the scientific and technological process and the complex patterns of communication and funding affect the existence and location of records. . . . For archivists this complex environment of internal and external associations requires comprehension of a universe of interconnected documentation. (p. 23)

  41. In 1989, Bearman, picking up where Haas, Samuels, and Simmons left off, challenged the whole approach to appraisal based on value concepts. Arguing that it is physically impossible to appraise the volume of records using traditional methodology and criteria and that no cost-benefit justification can withstand close scrutiny, he argued for a risk management approach to selecting records. The question to be asked is not "what benefits would derive from retaining records," but instead should be what is "the probability of incurring unacceptable risks as a consequence of disposing of records?" (p. 10). He further stresses the need to focus less on the records, but more on the activities they document:

    We will only be able effectively to appraise larger volumes of records if we focus our appraisal methods on selecting what should be documented rather than what documentation should be kept, and develop tactics for requiring offices to keep adequate documentation, rather than trying to review what they have kept to locate an adequate record. (p. 15)

  42. Bearman and Hedstrom (1993) suggest that "all archival activity focuses on the business activity of organizations, the requirements they present for accountability" (p. 97).

  43. Samuels (1992) argues that institutional functional analysis is the first step in all archival processes, including appraising. The functional approach "frees the observer from focusing on the particulars, such as the name of the office that created the record, and promotes greater understanding of the purposes for which the records were created" (p. 4). The focus needs to be on what the organization does rather than on who does it. Institutional functional analysis shifts the focus of appraisal from the record to the functional context in which the record was created (Cook, 1993). The analysis takes place on two levels, the generic and the specific. In the generic analysis, all the functions are treated equally. After this is accomplished, a specific institutional analysis is done where the functions are assigned values, which shift the appraisal focus. Functions are generally grouped into three broad categories: core functions, support functions, and sustaining functions (Cook, 1993).

  44. Cook (1992a) argues forcefully that the "concentration of archival appraisal on groups of records and on the search for 'value' in them may well be misdirected . . . [and] the focus of appraisal should shift from the actual record to the conceptual context of its creation" (p. 38). Using a macro-appraisal methodology, the conceptual purpose of the records is identified: ". . . macro-appraisal emphasizes . . . the archival value of the location or site or circumstances of records creation rather than the value of the records themselves. It assesses . . . in a global way rather than dealing directly . . . with individual records" (p. 53). He advocates a focus on the functions causing the creation of a record and on the structures affecting its creation rather than on the record itself.

  45. Sanford and Thiele (1996) have presented arguments in favor of a macro-functional approach to the appraisal of government records. Based on their experience with the Vermont Information Strategy Plan (VISP) project in Vermont, they were able to show that functional appraisal using business process analysis methodology was effective in identifying business functions and locating key collection sites. The problem was that the time and expense needed to do the analysis placed the techniques beyond the reach of most archival institutions. Instead they argued for a generalized macro-functional approach where broad areas of government activities are identified and categorized, and the appraisal process is used to acquire documentation in each of the functional areas based on weighted criteria. Modified forms of the business process analysis are used to identify business transaction collection sites.

  46. Cox (1994) notes that the major change in scope of archival appraisal in the United States is from an institutionally bound focus to a multi-institutional focus. The development of the documentation strategy model is a result of this change of focus. Cox identifies twelve archival appraisal principles, and argues in support of a documentation strategy model to guide the selection of records. Contending that the most important records are the ones that are used the most, and therefore the ones most likely to be lost or damaged, he argues that archives cannot wait until the end of the document life cycle to select records, but instead must become involved nearer the time of their creation. He further states that the documentation strategy concept can go a long way in meeting the representativeness issue raised by historian Yates. Cox also argues that archivists cannot depend on historians as a source of criteria for making judgments about long term value because the historians' identification of valuable documentary materials vary as the political and social winds change the focus of their research interests. Instead, the effort must be to provide some documentation of all the functions that government has engaged in.


    DOCUMENTATION APPRAISAL STRATEGY

  47. Documentation strategy grew out of efforts in the 1970s to document social movements, minority issues, and other issues not adequately covered in most archival collections. The concept was developed by the efforts of the Joint Committee on Archives of Science and Technology (JCAST), the Minnesota Historical Society, the State Historical Society of Wisconsin, and the Bentley Historical Library during the late 1970s and early 1980s. It was first defined at the 1984 meeting of the Society of American Archivists in a series of papers delivered by Samuels, Hackman, and Aronnson (Cox, 1990).

  48. The concept is a response to the growing complexity of society and the failure of traditional appraisal methodology to document adequately the activities, interrelations, and interactions of governmental bodies, private enterprises and individuals. To document even a simple activity or achievement requires the records of many organizations, government agencies, and individuals. Adding to this growing complexity has been the electronic revolution which makes it possible to create, manipulate, duplicate, modify, and erase records with considerable ease (Cox, 1990; Samuels, 1992).

  49. The documentation strategy model focuses on current activities or issues. It seeks to enlist the aid of record creators, administrators, historians, archivists, and other interested parties in identifying what areas of a business process or activity need to be documented and collected in order to ensure an adequate record is obtain. As a result of this analysis, efforts are made to ensure that adequate records are created and collected. The focus is less on what records exist and more on what records should exist. This analysis is used to classify where and which types of records are to be collected, and the physical examination of existing records is bypassed (Reed, 1993).

  50. Samuels (1992) argues that this approach seeks to reduce the subjective guessing that traditional appraisal uses and replaces it with clear documentation objectives that are based on a thorough understanding of the function, phenomenon, activity, or institution being documented. This approach seeks to document the entire spectrum of the activity rather than depend on the bits and pieces that make their way to the archives. It mandates an activist role for the archivist to ensure that adequate documentation is being produced at the critical record points. This is critical when dealing with non-print records. By being able to identify in advance the areas that need to be documented, efforts can be made to ensure that the appropriate steps are taken to ensure the upgrading and preservation of these records.

  51. Cox (1990) argues that the use of documentation strategies as appraisal strategies has emerged for the following reasons: 1) the traditional systems are decentralized, uncoordinated, reactive, passive and duplicative; and 2) in modern society, creation and use of information is spread among a diverse group of records creators. Documentation strategies allow a rational approach to dealing with the super-abundance of records and information.

  52. Samuels (1992) uses functional analysis to identify seven functional areas that university activities can be separated into. She identifies, weighs, and classifies activities the university community engages in. Further analysis allows her to select the critical areas that need to be documented and the type of records that need to be collected based on this classification. Identifying documentation that other repositories are collecting from her list, she is able to concentrate her efforts on classifying and collecting the necessary documentation to fill in the gaps.


    BUSINESS PROCESS ANALYSIS

  53. Business process or transaction analysis in general refers to the same process. A combination of systems analysis from computer science and business modeling from management, this process identifies all the activities involved in the performance of a business activity. It has proved to be a potent tool for restructuring business because it follows the business activity across organizational boundaries, and identifies the inputs and outputs at each stage of the activity. The key archival and records management concept behind process analysis is that records are created only when they are required to move the process forward or to document some transaction or activity. Appraisers first need to examine the business process being documented to determine whether or not archival type records are being produced. If the analysis indicates that the business process has the potential of producing large quantities of archival records, then more detailed examination of those records can be scheduled. If the analysis indicates little potential of archival records being created, then the business process can be placed low on the schedule for detailed examination (Reed, 1993).

  54. Business process analysis has an added benefit for the archivist. Because of its emphasis on business functions and activities and not on organizational structure, it enables an appraisal across structural lines. Functions which appear as minor activities of many agencies when examined using traditional methodologies, gain in importance when appraised across structural lines using business process models. Some functions that appear to be of major importance in traditional appraisal appear much reduced in importance when examined against the entire functional activity of an organization. The identification of common functions across structural lines allows the development of standardized classification and treatment of similar record types. These standardized approaches often show up in revisions of General Records Schedules. The General Records Schedules provide a unified classification structure for a wide variety of record types with built-in disposition recommendations.

  55. The business process analysis also allows the appraisal of functions that overlap organizations or agencies. Records in different organizations that deal with different aspects of the same function can be identified and appraised together (Reed, 1993). Rick Brown (1996) of the National Archives of Canada used immigration to illustrate the importance of this aspect of appraisal. He pointed out that no one agency in Canada has immigration as a major function. If appraisal is based on agency major functions, most records dealing with immigration would be destroyed without examination. With the business process analysis, all the offices dealing with immigration were identified, and the critical record collection points scheduled for examination.

  56. The most intensive effort at using business process analysis in government was undertaken by the State of Vermont. Using techniques developed by the James Martin Company, Vermont analyzed the executive branch. They identified seven major functions and distributed government agencies under those functions (Vermont. Government Consulting Group, 1992). Subfunctions were identified and further studies undertaken. The emphasis was placed in two areas: revenue collection and client services.

  57. The revenue collection appraisal grouped four major revenue collection or generating agencies together. The resulting report established a consolidated view of the revenue collection process, identified areas of major record keeping importance, and developed a unified terminology. The process went far enough to identify data entities and allow for the development of a new unified computer system devoted to revenue collection. The data tables provide much of the necessary metadata information needed to appraise the electronic records. The major problem with the study is that there was no input from the State Archives so that appropriate provenance metadata could be included in the system as a design feature (Vermont. Government Consulting Group, 1994). Even without the provenance data, for the first time in Vermont history there is a complete functional breakdown of the revenue collection function in state government.

  58. This study also showed how expensive it is to carry a business process analysis all the way down to the data element level. It took four full-time employees two years to perform the analysis. The State of Vermont did end up saving money because of this study, because it enabled the state to streamline the revenue collection activities into one computer system. The lesson for archivists is that they need to assist other agencies in their efforts to computerize their functions so that the archives can obtain this type of detailed information.


    MACRO-FUNCTIONAL ANALYSIS

  59. Macro-functional analysis, or macro-appraisal, is a top-down analysis of the functions and the record creator. The leading proponent of this approach is the National Archives of Canada. Faced with an explosion of records to be appraised and very limited resources, the National Archives concluded that the traditional appraisal methodology was totally inadequate. Opting for an approach that combined functional analysis with structural provenance the Canadians classified each agency into one of four categories based on its importance and likelihood to provide significant archival quality records.

  60. The criteria for classification of institutions under this approach were: 1) importance of institution within the government; 2) breadth and diversity of institution's mandate and functions; 3) size and complexity of institution; 4) identification of anticipated duplication of some function in other institutions; and 5) important archival records in several media or multi-media formats. Gaps in National Archives holdings or records at risk were not considered to be relevant criteria (Canada, 1990).

  61. The 174 government institutions were classified into four categories: 1) institutions with large impact on Canadian society or government and anticipated high archival payoff; 2) significant and separate institutions not covered by category 1; 3) smaller institutions with narrow focus; and 4) institutions of limited or marginal nature (Canada, 1990).

  62. With the failure of the VISP effort to reorganize information handling along functional lines in Vermont, the State Archives assessed the value of VISP as a macro-appraisal tool. It was recognized that the business process approach had high value for appraising government records on functional process lines, but was less successful as a macro-appraisal tool. Moving away from the traditional functional division of government into three co-equal branches, the State Archives sought to identify functions common to all levels and branches of government. They identified six major or macro-functional areas and were able to classify all the government structures across these macro-functions:

    1. address needs, responsibilities, and grievances;
    2. adjudicate offenses, disputes, and disagreements;
    3. cultivate inter- and intra-governmental relationships;
    4. manage resources (stewardship);
    5. provide security and protection; and
    6. promote the social contract.

    They discovered that this approach provided a framework that links the executive, legislative, judiciary, bureaucracy, and extra-governmental sources together. It also revealed previously unidentified relationships among the many units of government. The archives is developing agency histories to include these functional links as a classification tool (Sanford & Thiele, 1996).

  63. Alabama is also moving toward a macro-functional analysis of government as a way to gain control over records, especially electronic records. Using a process similar to one used in Vermont, the Alabama Government Records Division identified eight broad functions of government and created a matrix of the agencies performing each function:

    1. economic development;
    2. client services;
    3. regulation;
    4. stewardship;
    5. administrative support operations;
    6. public advocacy;
    7. policy and statute development; and
    8. law enforcement and emergency powers. (Alabama. Government Records Division, 1996a)

    Then by linking the record keeping warrants (i.e., statutes and regulations) with the transactions involved, appropriated retention schedules and requirements will be issued. Alabama also plans to identify transactional areas where records are not being created and mandate their creation. Based on this analysis, the archives will then be able to classify transactions and prioritize record appraisal and collection activities (Alabama. Government Records Division, 1996b).


    DISCUSSION

  64. The majority of the commentators dealing with appraisal, provenance, and the electronic or computer revolution agree that the traditional bottoms up approach is inadequate to handle the growing mass of records. The thousand or so professional archivists in the United States and Canada are unable to control this approaching tsunami. To add to the problems, the electronic revolution has changed many of the ground rules.

  65. The electronic revolution began in earnest with the development of simple copying technology--the old thermofax machines followed by the wet toner copiers and finally the dry toner copiers. These machines made it possible for multiple copies of records to be produced and distributed across the records landscape. Where earlier archivists had to deal with a single copy of a record located in one office, current archivists are now faced with redundant copies, often running into the hundreds, distributed among multiple offices. Still, there was the potential to manage this with the traditional approach, because these print-based copies still had most of the provenance signatures attached to them.

  66. Electronically-based media has changed this forever. Unlike print-based media (paper, parchment, and vellum) which has an expected life span running into the hundreds of years, the electronic media (audio tape, video tape, magnetic tapes and diskettes, optical discs, etc.) has a hoped-for life expectancy of 20 years. Experience shows that the life expectancy of the denser storage media is closer to 5 years or less. The realization that electronic records are unstable to the point that they might disappear before they can be appraised has galvanized the archival profession into examining its core assumptions.

  67. The first assumption to be addressed was that records were appraised at the end of their life cycle and classified into two major categories: records that were to be retained and maintained for some specified period of time, and everything else that was to be disposed. The likelihood that records recorded on electronically-based media would not survive long enough to reach the appraisal stage indicated that the appraisal process needed to be moved forward toward the beginning of the record life cycle. Many authors have argued that appraisal should take place at the moment of creation: "Now is the time for archivists to be present at the creation of documents, to ensure that they are designed not only to serve immediate administrative ends but also administrative/historical research for policy planning and development" (Taylor, 1984, p. 30). As Dearstyne (1993) points out, "this type of advocacy . . . has brought counter appeals for caution and adherence to traditional missions" (p. 229). Most of the driving force for moving the appraisal process toward the beginning of the record life cycle is coming from archivists at major institutions and archives where they are feeling the brunt of the growing record mass.

  68. The second assumption to be called into question is Schellenberg's (1956) approach of evaluating records based on their evidential or informational value. Schellenberg favored physical examination of the records or use of sampling techniques to make appraisal decisions. Under this approach, the decisions were based on the contents of the records themselves. The general understanding that emerged during the 1980s and 1990s is that this approach is unworkable with both print-based and electronic records. There is no way to examine physically over a trillion documents each year to make appraisal decisions.

  69. A consensus has emerged that the most efficient way to appraise and classify this growing mass of records is to use some form of functional analysis. Functional analysis allows the archivist to identify critical business functions that need to be documented. Using the results of this analysis, classification categories can be established that allow the placement of record groups into different classes. Several different approaches have been suggested, including the Canadian, Vermont, and Alabama models.

  70. The approach furthest along in its implementation is the Canadian model. The Canadian model breaks with archival theory and differs from most efforts in that it classifies the institutions based on their potential to produce archival quality records, not the records themselves. This process classifies most Canadian institutions into one of four categories using a series of functional requirements and value judgments. The argument is made that institutions create the records and the National Archives of Canada must deal with institutions, not with records. Priorities will be determined by detailed research on history, functions and mandates of the institutions, information systems, and profiles involved (Canada, 1990).

  71. Both the Vermont and Alabama models make use of macro-functional analysis to identify major functional areas. They then use business process modeling to identify processes that are likely to produce archival quality records, and also locate junctures where the probability is highest that such records will be produced. After mapping the governmental structures against the functional process model, they are able to classify institutional locations where there is a high probability of quality records being located.

  72. These functional process models are interested in documenting the fact that some transactional process or activity has occurred, and are less interested in the contents of that interaction. They view government as a process, and see records as objects documenting the workings of the process. According to this view, the only record content of importance, in an archival sense, are the records where the process broke down or encountered difficulties because they showcase the full workings of the process under pressure.

  73. The third assumption shattered by the electronic revolution was that "the container is the record." Before the electronic revolution, the print-based document containing the recorded details of the transaction was the record; they were inseparable. Great weight was placed on the form and structure of the record. With the advent of the electronic record all this has changed. The record is no longer tied to its container. An electronic document exists as scattered data bits throughout the memory storage unit of the computer. These bits can be reassembled into a wide variety of presentation forms, all of which contain the recorded information. This has led to the development of transaction based definitions of records which is having a major impact on appraisal and record classification.

  74. One of the goals of appraisal is to separate records from nonrecords. The transactional based records definitions require a deeper understanding of the functions, processes and interrelationships among record creators in order to achieve this aim than the traditional approaches do. These transactional based definitions identified a third class of electronic information objects--the information database. The information database or data warehouse contains information that has value and should be appraised for continuing retention. It is not a record, nor is it a nonrecord.

  75. The fourth assumption that needed modification was the Principle of Provenance. In the print-based world, the physical copy of the record was encrusted with provenance signatures. The office of the record creator was identified, the official responsible, date, location, and so forth. In addition, the office and filing system where the record was stored also provided detailed provenance information that aided in the appraisal and classification of the records in question. Even the copying machine did little to disrupt this system. The electronic record shattered the tie between the record and its provenance.

  76. What the electronic record requires is the active introduction of provenance signatures to the record. They can no longer be allowed to just happen. One of the motivations behind the movement to functional analysis as part of the appraisal process was to begin tying context (i.e., provenance) back to the record. By linking the business process and its associated structural members to the record, provenance was restored to the appraisal equation. The development of detailed institutional histories which are linked to functional and process descriptions provided needed information to aid the classification process. These histories also act as authority files which track the movement of functions and related business processes among different organizational units. All the archives using the functional approach to appraisal are developing these histories in considerable detail (Alabama, 1996; Canada, 1990; Sanford & Thiele, 1996).

  77. The last assumption to be discussed is the importance of the Principle of Original Order. Considered to be the internal portion of respect des fonds, it has not fared well. Within the print-based world, the original order could be determined from the filing rules and the physical presence in filing cabinets. Appraisal and classification could make use of the order and arrangement developed and used by the originating office. All this is impossible in the electronic office.

  78. In the electronic office data bits cannot be examined to determine original order. These bits are strewn randomly throughout the electronic storage space by the memory management system. The memory management system continually reorganizes the structure of the memory spaces to maximize the storage space. In systems working with a data warehouse, networked or not, there are many links to data items in the warehouse. There is no simple way to deduce original order or filing structure from the electronic office.


    CONCLUSIONS

  79. The computer and its associated networking technology is putting severe stress on archives and appraisal methodology. The computer, with its ease of duplication, has multiplied the number of records. Network technology has enabled copies of records to be held in computer databanks all over the world. All this has had a profound effect on appraisal.

  80. The first effect has been upon the concept of the record. Since the data elements of the record are scattered throughout the electronic media and are reassembled into a variety of formats through the use of external programs, the record has been disassociated from its container. Also, the record has become a distributed entity, focusing on the shift from the contents of the record to the transactions for which the records are evidence.

  81. Appraisal is the key to handling records, especially in the electronic world. Less than 1% of the content in records justifies preservation, so appraisal and classification systems that capture the context and process most of the institutional activity will be sufficient. Appraisal needs to be shifted towards the moment of record creation.

  82. Macro-functional analysis has the most potential for becoming an effective appraisal and classification tool. Properly done, it ensures representative documentation across institutional structures and also identifies functions that cross structural boundaries. Because it focuses on functions and processes and requires good institutional histories as part of the process, it promotes the development of efficient and effective classification schemes.

  83. Business process analysis describes the inner workings of the process in question, and also identifies the entities and flow patterns involved. It locates critical junctures which allow focused appraisal and collection activities, but has the severe drawback of being a very labor intensive process. Archivists are going to have to piggyback their efforts onto information reengineering activities being done for other purposes in order to acquire this knowledge.

  84. Provenance is still important. A strong case can be made for it becoming more important in identifying the context surrounding electronic records. A major effort will be required to link provenance information to the record as it is created in the electronic environment, since this type of linkage needs to be part of the system design. This will require the development of sophisticated metadata systems to ensure that this critical information is systematically attached to the records as they are created. In order to ensure the proper appraisal and classification of records, archivists and records managers are going to have to move to the forefront of their institution's record creation activities.

  85. The burgeoning mass of organizational records and development of the electronic records environment is forcing archivists and records managers to re-examine their basic assumptions. The electronic environment has disassociated the record from its print-based container. The record needs to be redefined in functional terms. This disassociation has also increased the need for provenance signatures that are attached to each record. The traditional bottoms up approach to appraising records based on record contents is unworkable in the face of the ever increasing numbers and complexity of modern records. Archivists and records managers need to use top down functional approaches that ensure adequate documentation of organizational activities and functions. They also need to appraise records nearer the moment of their creation.


REFERENCES

Alabama. Department of Archives and History. Government Records Division. (1996a). Functional analysis of Alabama government. Montgomery, AL: Government Records Division.

Alabama. Department of Archives and History. Government Records Division. (1996b). State government records program: Defining a new records appraisal strategy. Montgomery, AL: Government Records Division.

Bearman, D. A. (1989). Archival methods (Archives and Museum Informatics Technical Report No. 9). Pittsburgh, PA: Archives and Museum Informatics.

Bearman, D. A. (1994). Electronic evidence: Strategies for managing records in contemporary organizations. Pittsburgh, PA: Archives and Museum Informatics.

Bearman, D. A., & Hedstrom, M. (1993). Reinventing archives for electronic records: Alternative service delivery options. In M. Hedstrom (Ed.), Electronic records management program strategies (pp. 82-98, Archives and Museum Informatics Technical Report No. 18). Pittsburgh, PA: Archives and Museum Informatics.

Bearman, D. A., & Lytle, R. H. (1985). The power of the principle of provenance. Archivaria, 21(Winter), 14-27.

Bellardo, L. J., & Bellardo, L. L. (1992). A glossary for archivists, manuscript curators, and records managers. Chicago: The Society of American Archivists.

Boles, F., & Young, J. M. (1991). Archival appraisal. New York: Neal-Schuman Publishers, Inc.

Brown, R. (1996). Macro-appraisal theory. Paper presented at the 1996 Annual Meeting of the National Association of Government Archives and Records Administrators (NAGARA), Washington, D.C. July 17-20, 1996. Session #12: Macro Appraisal in Practice.

Brunton, P., & Robinson, T. (1993). Arrangement and description. In J. Ellis (Ed.), Keeping archives (2d ed., pp. 222-247). Port Melbourne, Victoria, Australia: D. W. Thorpe in association with The Australian Society of Archivists.

Burke, F. G. (1988). Commentary [to Cox & Samuels (1988)]. American Archivist, 51, 47-51.

Canada. (1990). National Archives of Canada Act. Government-Wide Plan for the Disposition of Records, 1991-1996, 22 October 1990. Ontario: Disposition Coordinating Committee.

Canada. (1991). National Archives of Canada Act. Report of the Multi-Year Records Disposition Plan Work Group, March 7, 1991. Ontario: Disposition Coordinating Committee.

Canada. (1992). Management of Recorded Information, Directive 7-5, June 1992. Ontario: National Archives of Canada.

Cook, T. (1992a). Mind over matter: Towards a new theory of archival appraisal. In B. L. Craig (Ed.), The archival imagination: Essays in honor of Hugh A. Taylor (pp. 38-70). Ottawa: Association of Canadian Archivists.

Cook, T. (1992b). The concept of the archival fonds: Theory, description, and provenance in the post-custodial era. In T. Eastwood (Ed.), The archival fonds: From theory to practice (pp. 31-85). Ottawa: Bureau of Canadian Archivists.

Cook, T. (1993). Taming the information behemoth: Institutional Functional analysis as a tool for appraising the records of modern society. Draft. 5 November 1993. (Typescript sent to the VISP Project, Vermont State Archives, by T. Cook in 1994).

Cox, R. J. (1990). American archival analysis: The recent development of the archival profession in the United States. Metuchen, NJ: Scarecrow Press.

Cox, R. J. (1994). The documentation strategy and archival appraisal principles: A different perspective. Archivaria, 38(Fall), 11-36.

Cox, R. J., & Samuels, H. W. (1988). The archivist's first responsibility: A research agenda to improve the identification and retention of records of enduring value. American Archivist, 51(Winter/Spring), 28-42.

Dearstyne, B. W. (1993). The archival enterprise: Modern archival principles, practices, and management techniques. Chicago: American Library Association.

Dollar, C. M. (1992). Archival theory and information technologies: The impact of information technologies on archival principles and methods (Informatics and Documentation Series, 1st ed.). Macerata, Italy: University of Macerata.

Eastwood, T. (1992). General introduction. In T. Eastwood (Ed.), The archival fonds: From theory to practice (pp. 1-14). Ottawa: Bureau of Canadian Archivists.

EPA. (1994). Summary of current records management policy on electronic mail. Attachment A of Notice 88-1 Safeguarding and Disposition of Official Records (Draft). Handout from Working Meeting on Electronic Records Management, Pittsburgh, PA. April 8-10, 1994.

Haas, J. K., Samuels, H. W., & Simmons, B. T. (1985). Appraising the records of modern science and technology: A guide. Cambridge, MA: Massachusetts Institute of Technology Press.

Ham, G. F. (1993). Selecting and appraising archives and manuscripts. Chicago: Society of American Archivists.

Lutzker, M. A. (1982). Max Weber and the analysis of modern bureaucratic organization: notes toward a theory of appraisal. American Archivist, 45(2), 119-130.

McKemmish, S. (1993). Introducing archives and archival programs. In J. Ellis (Ed.), Keeping archives (2d ed., pp. 1-24). Port Melbourne, Victoria, Australia: D. W. Thorpe in association with The Australian Society of Archivists.

Menne-Haritz, A. (1994). Appraisal or documentation: Can we appraise archives by selecting content? American Archivist, 57(Summer), 528-542.

Minnesota. (1994). Minnesota statutes annotated (Volume 10B, Sections 127 to 143). St. Paul, MN: West Publishing.

Murphy, C. (1996). Backlogs of history. The Atlantic Monthly, 277(5), 20-22.

National Archives and Records Administration. (1993). A Federal records management glossary (2d ed.). Washington, DC: National Archives and Records Administration.

North Carolina. (1995). General statutes of North Carolina. Chapter 132, Public Records. Charlottesville, VA: Michie.

Public Printing and Documents Act, 44 U.S.C. § 3301 (West 1991).

Reed, B. (1993). Appraisal and disposal. In J. Ellis (Ed.), Keeping archives (2d ed., pp. 157-206). Port Melbourne, Victoria, Australia: D. W. Thorpe in association with The Australian Society of Archivists.

Robek, M. R., Brown, G. F., & Maedke, W. O. (1987). Information and records management (3d ed.). Encino, CA: Glencoe Publishing.

Roberts, D. (1994). Defining electronic records, documents and data. Archives and Manuscripts, 22(1), 14-26.

Samuels, H. W. (1992). Varsity letters: Documenting modern colleges and universities. Metuchen, NJ: Society of American Archivists and Scarecrow Press.

Sanford, G. D., & Thiele, Jr., H. E. (1996, July). Government macro-functions. Paper presented at the 1996 Annual Meeting of the National Association of Government Archives and Records Administrators (NAGARA), Washington, DC.

Schellenberg, T. R. (1956). The appraisal of modern public records. Washington, DC: U.S. Government Printing Office.

Standards Australia. (1995). Records management. Part 1: General requirements (DR 95194). Strathfield, New South Wales, Australia: Standards Australia.

Taylor, H. (1984). Information ecology and the archives of the 1980s. Archivaria, 18, 30.

Toffler, A. (1990). Power shift: Knowledge, wealth, and violence at the edge of the 21st century. New York: Bantam Books.

Vermont. (1995). Vermont Statutes Annotated. Title 1 through Title 3. Charlottesville, VA: Michie Publishers.

Vermont. Government Consulting Group. (1992). Vermont Information Strategy Plan. Montpelier, VT: Government Consulting Group.

Vermont. Government Consulting Group. (1994). Revenue Collection Information Strategy Plan. Montpelier, VT: Government Consulting Group.