ALLERTON 1996
Breakout Session
Monday, October 28, 11:30-12:30PM

Developing a Framework for Evaluating Digital Libraries

Presented by

Gary Marchionini
University of Maryland
march@oriole.umd.edu

Note: Below is a transcription of the overheads created during this breakout session 
Theses and Dissertations

    Other Projects

  Processes/Approaches

    Theories/Models

      \        / 
       \      /
        \    /

       FRAMEWORK
TOP

Theories

Aspects of DLs

Evaluation Process

THEORIES

Sense-Making

Work Practice

Lifecycle

DL IN CONTEXT

--> Content
--> Using
--> System
--> Management

CONTENT

Who/how creates/ed
Who/how selects/ed
Who/how organizes/ed

(For these, context is related to goals and policies)

Why
How understandable
Coverage -- completeness, depth
Bib/fulltext/multimedia -- representations
Metadata, catalog, finding aids
Taylors dimensions: accuracy, currency, validity

USING (depends on theory -- those organize, include/exclude) 
Search   --   Value-added
Retrieving    Share
Find          Communicate
Discover      Copy
Browse        Republish
Extract
Create
Store
Display
Sort
Organize
SYSTEM 
Functionality   --   System-dependent
Interface            Openness
Speed                Durability
Reliability          Where can be accessed from
Robustness           Maintenance
User-orientedness    Staff level, skills, number
Algorithms           Capacity
Uncertainty handling Security
Scalability
Expense
Extensibility        (Performance)
SERVICES

Reference
Training
Translate
Publish
Copy
Summarize
Deliver
Financial Management
IPR Management
Help
Bibliographies
Filter
SDI
Indexing
Cataloging
Perserving
Achiving
Annotations

COMMUNITY

Who, where
Job profiles
Location

Organization: 

Work habits
Cohorts
Institutions
Goals
Cultures   <----------------
Funders                    |
Size           
Centralized/Decentralized  |
Norms   
Memory                     |
Profitability
                           |
Collaboration:
                           |
Coordination
Interaction                |
Communication
Competition 
Proximity                  |
Critical mass
Conflict                   |
Conflict resolution
Relationships              |

Users:                     |

Empowered                  |
Goals
Needs                      |
Motivation
Stakeholders   -------------     
Roles/relationships --> Benefactors
Memory
MANAGEMENT

Control
Open/closde
Accountability
Responsiveness
Flexibility
Cost
Staff needed
Dynamics
Changeability
Overhead
Policies: Access, Charging
Planning
Risk Management
Options
Resources
Style (e.g., hierarchical)
Structure
Centralized/decentralized
Status
Pay
Authority

PURPOSE

Compare, improve systems, design process
Promote use, acceptance
Understand impact
Understand viability
---cost effectiveness
---DLs
Satisfy management (accountability)
Carry out: IR, HCI, Hypertext, design, DB, MM ... research
Check goals, usability
Verify, validate design/requirements

GOALS

Evaluate if service, use, etc. (or, indicators/metrics for service and use, etc.) are met/supported

OUTCOMES 
Performance
  Purpose specific
    --Learning

Knowledge/technology transfer
  Adoption
  Adaption

---------
Viability
Rewards and punishments
Participation
Profits/benefits
---------

Goal migration      |  Series of failures accumulating
Change in practice  |

Meta issue: levels

Matrix (from purpose, goals, objectives)
  Or tree?
  Correlations?
MEASURES

Acceptance of DL by organization with respect to goals
Acceptance by users with respect to goals
Effectiveness of system with respect to tasks
Ability of users to understand system during use (fn: knowledge, ...)
Traffic
No. of users (people, institutions)
No. of searches, links
Time on task, errors
Substitution effects (e.g., for paper)
Complementary effects
Repeated use
Satisfaction
Amount of help needed, training needed
Unintended uses/serendipity
Time sinks (where time gets used, wasted)

PROCESS

Time
Instruments to help
Tutorials on ...

(The above three items are needed for various approaches)

Sustainability

Predict/test
---For various time spans
---On users, organizations

MATRIX FOR EVALUATING EDUCATIONAL IMPACT OF A DL FROM MANY VIEWS 
TIME:
     20-50 yrs.    |
     (very long)   |
                   |
     10-20 yrs.    |
     (long term)   |
                   |
     5-10 yrs.     |
     (med. term)   |
                   |
     3-5 yrs.      |
     (short term)  | 
                   |
     year          |
                   |
     semester      |
                   |
     session/day   |
                   |
                   ---------------------------------------------->

     STAKEHOLDER:  Indiv. -> Small group -> Large group -> Society

           (e.g., student -> class -> school -> field -> world)

Allerton 1996 Index

Last Updated: Feb. 11, 1997