Quality Modeling

Digital Libraries FoundationsWorkshop at JCDL 2007

Vancouver – June 23

Edward A. Fox

Virginia Tech, USA

fox@vt.edu http://fox.cs.vt.edu

Acknowledgements: Students

• Ghaleb Abdulla, Robert France, Marcos Gonçalves, Doug Gorton, Rohit Kelapure, Neill Kipp, Aaron Krowne, Yi Ma, Paul Mather, Uma Murthy, Ananth Raghavan, Unni. Ravindranathan, Ryan Richardson, Rao Shen, Ricardo da Silva Torres, Srinivas Vemuri, Seungwon Yang, Qinwei Zhu, …

Acknowledgements: Faculty, Staff

• Lillian Cassel, Lois Delcambre, Debra Dudley, Joanne Eustis, Weiguo Fan, James Flanagan, Sandy Grant, Eric Hallerman, Douglas Knight, Alberto Laender, David Maier, Manuel Perez-Quinones, Layne Watson, …

Other Collaborators (Selected)

• Brazil: FUA, UFMG, UNICAMP

• Case Western Reserve University

• DELOS and its many members

• Emory

• Germany: Univ. Oldenburg

• Portland State University

Acknowledgements: Support

ACM, AOL, CAPES, CNI, DFG, IBM, IMLS, Microsoft, NDLTD, NLM, NSF (IIS-9986089, 0080748, 0086227, 0307867, 0325579, 0535057, 0535060; ITR-0325579; DUE-0121679, 0121741, 0136690, 0333531, 0333601, 0435059, 0532825, …), SUN, …

Quality

• Big picture

• Quality of union DLs

• 5SQual

• Prior Work on Measuring DL Success

• IT/ IS success/adoption

• DL success model

• DL success, info, seeking/utilization

Quality Big Picture

• Purpose, Goals, Objectives– Advancing scholarship/civilization, Communication,

Sharing/use, Open access/preservation, …• Perspectives, Observations

– User/patron/personal, organizational, community– System Performance

• Metrics, Measurements• Models• Analysis, Reporting• Continuous Improvement (logging -> enhancement)

• See also next talk for literature review

Asynchronous, Digital Library Mediated (Scholarly) Communication

Different time and/or place

Digital LibrariesShorten the Chain from

Editor

Publisher

A&I

Consolidator

Library

Reviewer

To a flatter, service-oriented, more general framework

Author

Reader

Digital

LibraryEditor

Reviewer

Teacher

Learner

Librarian: designing, acquiring, organizing, collaborating, evaluating

Machine

Context in which the DL is embedded

Quality dimensions and measurement for minimal union DLs

Union DL Concept within the 5S Framework

Quality dimension Measurement

Union Catalog

Consistency# of sets of metadata specifications per digital object

Completeness

# of digital objects without a set of metadata specifications; size of the described collection

Union Service(mapping, searching, browsing)

Mapping-completeness

# of local schema elements which have been mapped to global schema

Coverage

# of results retrieved from a union DL; # of total results retrieved from the DLs to be integrated

NavigationGain

# of navigation paths available in a union DL; # of navigation paths available in the DLs to be integrated

5S definitional structure

Digital Object

RepositoryRepositoryCollectionCollection Minimal DL

Metadata Catalog

Descriptive Metadata

Specification

Structural Metadata

Specification

StreamsStreams StructuresStructures SpacesSpaces ScenariosScenarios SocietiesSocieties

indexingindexing

browsingbrowsing searchingsearching

servicesservices

hypertexthypertext

Structured Stream

Digital Objects

Metadata

Services

• Completeness

• Conformance

• Accessibility

• Similarity

• Significance

• Timeliness

• Efficiency

• Reliability

Numeric

Indicators

5SQual - Dimensions

5SQual Archi-texture

Evaluations – Charts

has an exampleIS success model

Prior Work on Measuring DL Success

DeLone et al.

Seddon

has an exampleInformation seekingbehavior model

Ellis

Kuhlthau

has an exampleDL quality model Gonçalves

has an exampleUsability of DLsinspection of NCSTRL

evaluation of ACM, IEEE-CS,NCSTRL, and NDLTD

evaluation of ADL

evaluation of ADEPT

has an indicator

system usage

has an exampleTechnology

acceptance model Venkateshpredict Intention to re/use

Behavioral Attitude & Intension to Use — Venkatesh Model of IT Adoption

intention to use system

performance expectancy

system usage

Unified Theory of Acceptance and Use of Technology

effort expectancy

social influence

facilitating conditions

• Performance expectancy: – perceived usefulness, extrinsic motivation, job-fit, relative advantage,

and outcome expectations

• Effort expectancy– the degree of ease associated with the use of system

• Social influence– Subjective norms, social factors, and image

• Facilitating conditions– the degree to which an individual believes that an organizational and

technical infrastructure exist to support the system

Venkatesh Model of IT Adoption

DeLone and McLean Model of IS Success

System Quality

Information Quality

UserSatisfaction

IndividualImpact

OrganizationImpact

Use

Seddon Model of IS Success

System Quality

Information Quality

UserSatisfactions

Net Benefits to:

IndividualsOrganizations

Society

PerceivedUsefulness

DL Success Model

5S and minimal DLSynthesize

IS success and adoption models (see above) Information life cycle model (Borgman et al.) 5S-based DL quality model (Gonçalves et al.) Information-seeking behavior models (Ellis’ and

Kuhlthau’s)

From end user perspective

Information Life Cycle

AuthoringModifying

OrganizingIndexing

StoringRetrieving

DistributingNetworking

Retention/ Mining

AccessingFiltering

UsingCreating

crea

tion

distributionsee

kingutilization

search

ing,

browsing,

recommending

storing, archiving,

networkingau

thor

ing,

mod

ifyin

g,

desc

ribin

g org

anizi

ng, i

ndex

ing

pres

erva

bilit

y,

sim

ilari

ty,

tim

elin

ess,

accuracy, completeness,

conformance accessibility, preservability

pertinence,significance,

timelinessre

levan

ce

Activ

e Semi-active

Inactive

accessibility,accessing,

filteringGonçalves et al.

system quality (SQ)

performance expectancy (PE)

society, scenario,

space

service

accessibility

reliability

ease of use

joy of use

composabilityefficiency

effectiveness

extensibility

reusability

reliability

accessibility accuracy

completeness consistence conformance

pertinence preservability

relevance significance

similarity

timeliness

DL quality

dimension

information quality (IQ)

stream, structure

digital object

metadata

collection

catalog

repository

adequacy

relevance

reliability

scope

timeliness

understandability

social influence (SI)

DL success construct

society

5S and

DL concept

DL visibility

DL success

manifest variable

system quality (SQ)

performance expectancy (PE)

society, scenario,

space

service

accessibility

reliability

ease of use

joy of use

composabilityefficiency

effectiveness

extensibility

reusability

reliability

accessibility accuracy

completeness consistence conformance

pertinence preservability

relevance significance

similarity

timeliness

DL quality

dimension

information quality (IQ)

stream, structure

digital object

metadata

collection

catalog

repository

adequacy

relevance

reliability

scope

timeliness

understandability

social influence (SI)

DL success construct

society

5S and

DL concept

DL visibility

DL success

manifest variable

crea

tion

distributionsee

kingutilization

E1:starting

E2: chaining

E3: browsin

g

E4: diffe

rentia

ting

E5: monito

ring

E6: extra

cting

storing, archiving,

networking

K1:

in

itia

tion

K2:

se

lectio

n

K3:

explorationK4:

formulation

K5: collection

K6:

presentation

auth

orin

g, m

odify

ing,

desc

ribin

g org

anizi

ng, i

ndex

ing

pres

erva

bilit

y,

sim

ilari

ty,

tim

elin

ess,

accuracy, completeness,

conformance accessibility, preservability

DL Success Constructs

Activ

e Semi-active

Inactive

E: Ellis’ modelK: Kuhlthau’s model

satisfaction

performanceexpectancy

(PE)

behavioralIntention to

(re)use

relevance adequacy timeliness

reliability understandability scope

accessibilityease of use

joy of use reliability

userinterface

social influence (SI)

information quality(IQ)

system quality(SQ)

DL Success Model

information quality(IQ)

DL Concepts Regarding Information

collection metadata catalog

digital object metadata specification

repository

consist of

associated with

associated with

consist of consist of

accessibilitypreservability

pertinencerelevancesimilarity

significancetimeliness

digital object

accuracycompletenessconformance

metadata specification

impact factorcompletenesscollection

consistencycompletenesscatalog

consistencycompletenessrepository

understandabilityrelevancetimelinessreliabilityadequacy

digital objectmetadata specification

adequacy

catalogcollection

scope

repository

Information Quality (IQ)

Digital Library IQ

screen designnavigation

user interface

composabilityreusability

extensibilityefficiency

effectivenessreliability

serviceaccessibility

reliabilityease of usejoy of use

System Quality(SQ)

performanceexpectancy

Digital Library SQ

DL success

Construct

seeking phase utilization phase

starting (E1/K1)

selection exploration

(E2-E6)/(K2-K3)

formulation (K4)

collection (K5)

presentation (K6)

social influence DL visibility

information quality adequacy, scope accuracy

system quality ease of use

joy of use (interface)

accessibility accessibility accessibility

performance

expectancy

usefulness

(interface)

DL Success Constructs Associated with Seeking and Utilization Phases

DL Success Constructs Associated with Seeking Phase

E1: Starting’ activity in Ellis’ model (K1: ‘initiation’ stage in Kuhlthau’s model) Social Influence (SI) — DL visibility

Publicize existence of a DLProvide a DL alert service

DL Success Constructs Associated with Seeking Phase

E2-E6: ‘chaining’, ‘browsing’, ‘differentiating’, ‘monitoring’, and ‘extracting’ in Ellis’ model (K2-K3: ‘selection’ and ‘exploration’ stages in Kuhlthau’s model) Information Quality (IQ) System Quality (SQ) Performance Expectancy (PE)

DL Success Constructs Associated with Utilization Phase

K4-K6: ‘formulation’, ‘collection’, and ‘presentation’ stage in Kuhlthau’s model Information Quality

information accuracy information accessibility

Summary: Quality

• Big picture

• Quality of union DLs

• 5SQual

• Prior Work on Measuring DL Success

• IT/ IS success/adoption

• DL success model

• DL success, info, seeking/utilization

Summary: Quality Big Picture

• Purpose, Goals, Objectives– Advancing scholarship/civilization, Communication,

Sharing/use, Open access/preservation, …• Perspectives, Observations

– User/patron/personal, organizational, community– System Performance

• Metrics, Measurements• Models• Analysis, Reporting• Continuous Improvement (logging -> enhancement)

• See also next talk for literature review

Discussion