Date post: | 21-Jan-2015 |
Category: |
Technology |
Upload: | sebastian-kruk |
View: | 1,579 times |
Download: | 2 times |
Chapter Copyright 2007 Digital Enterprise Research Institute. All rights reserved. www.deri.org
Digital Enterprise Research Institute www.deri.ie
Evaluation of Semantic and Social Technologies for Digital Libraries
Sebastian R. Kruk, Ewelina Kruk, Katarzyna Stankiewicz
Digital Enterprise Research Institute www.deri.ie
Outline of presentation
What are Semantic Digital Libraries Overview of JeromeDL Semantic and social information discovery solutions Evaluation goals and setup Evaluation results Conclusions
2
Digital Enterprise Research Institute www.deri.ie
Semantic Digital Libraries - Motivation
• How to integrate and search information from different sources? • How to share and interconnect knowledge among people?
3
Digital Enterprise Research Institute www.deri.ie
JeromeDL - Semantic Digital Library
Open source prototype of Semantic DL research; jointly developed by Gdansk University of Technology, Poland and DERI
Social Semantic Information Spaces Semantic description (interconnected metadata)
Annotations provided by users (social metadata)
Collaborative search and browsing (interface)
Features Search and browsing based on semantics empowers users
Users contribute to the classification process
Users can understand community driven annotations
Users enhance digital content using blogs, wikis on the side
Library can interact with other Internet services
4
Digital Enterprise Research Institute www.deri.ie
9
Metadata and Services in JeromeDL
5
Digital Enterprise Research Institute www.deri.ie
Search and browse on semantics
Natural language templates allows to perform complex queries using natural language
Dynamic Collections easily extensible
Resource-based Recommendations customizable view of recommendations
TagsTreeMaps clustered tags rendered with treemaps algorithm
zoomable interface paradigm
MultiBeeBrowse collaborative and adaptive approach to perform complex browsing
Exhibit (SIMILE, MIT) powerful faceted filtering
6
Digital Enterprise Research Institute www.deri.ie
Natural language templates
Find articles related to mission in the context of aerospace
...QueryTemplates(Regular Expressions)
English Portuguese
Aerospacemission
skos:relatedskos:narrower
resultsmarcont:hasKeyword marcont:hasDomain
SELECT * FROM ....
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
...
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
A
C
D
E
F
Step 1: Find similar resources
G
...
E
C
B
A
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
AE
Step 1: Find similar resources
Step 2: Rank and filter according to user’s settings
...
EBA
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
AE
Step 1: Find similar resources
Step 2: Rank and filter according to user’s settings
...
EBA
summary (max. 3)
Digital Enterprise Research Institute www.deri.ie
Browsing on semantics
TagsTreeMaps filtering based on clustered tags
using treemaps to present the tag space
zoomable interface paradigm
MultiBeeBrowse collaborative browsing
allows to perform complex browsing operations
user can overview browsing context and look up browsing history
Exhibit (SIMILE, MIT) powerful faceted filtering
9
Digital Enterprise Research Institute www.deri.ie
TTM - Filtering with hierarchical tags
Problems with Tag Clouds: information overload (for large tag clouds)
cannot carry structure and/or semantics
querying model: only conjunctive queries
Solution: limits the information overload
– clustering tagging space
– limiting popularity range
zoomable browser on the tagging space
selecting multiple tags– fulltext filtering - easy highlight matching tags
– optional conjunctive (AND) and union (OR) mode
defined interfaces for delivering processors in the pipeline (e.g., clustering, filtering, coloring)
10
Digital Enterprise Research Institute www.deri.ie
TTM - Filtering with hierarchical tags
11
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Adaptive
Presenting results human-readable names of concepts
type-specific rendering
limiting information overload with stretch-text
Refining queries in-situ each concept is seed to new query
different actions based on concept type
Suggesting properties and concepts most frequently used
recently used
Accessible predicated names human-readable names of properties
support for inverted properties
12
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Collaborative
Nowadays people share: photos, music, links, etc. - why not queries ?
Collaborative filtering solution adapted for sharing browsing experience based on Social Semantic Collaborative Filtering service
users can tag/annotate their queries
users can share queries with their friends
13
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Zoomable
Helping users with different problems Finding results
Going back and forth in the refinement process
Overview of current browsing context
Replaying previous queries
4 views: Basic browsing view
Structured history view
HoneyComb view
Life-long history view
14
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
Digital Enterprise Research Institute www.deri.ie
Social Services in JeromeDL
• Involve users into sharing knowledge
– Blogs – comments and discussions about documents and
resources
– Tagging – collaborative classification
– Wikis – collaboratively edited additional descriptions, such as
summaries and interesting facts
• Preserve knowledge for future use
– users can learn from experience of others instantly
– recommend new, interesting resources based on users’ profiles
17
Digital Enterprise Research Institute www.deri.ie
What is Social Semantic Collaborative Filtering?
• Goal: to enhance individual bookmarks with shared knowledge within a community
• Users annotate catalogues of bookmarks with semantic information taken from DMoz or WordNet vocabularies
• Catalogs can include (transclusion) friend's catalogues
• Access to catalogues can be restricted with social networking-based polices
• SSCF delivers:– Community-oriented, semantically-rich taxonomies
– Information about a user's interest
– Flows of expertise from the domain expert
– Recommendations based on users previous actions
– Support for SIOC metadata
18
Digital Enterprise Research Institute www.deri.ie
knows
include
bookmark
Social Semantic Collaborative Filtering
19
recommend
Digital Enterprise Research Institute www.deri.ie
Scope of Evaluation
20
Digital Enterprise Research Institute www.deri.ie
Questions for Evaluation
Question 1: Do the social and semantic services increase the quality of the answers provided by the users in response to given problems?
Question 2: Do the social and semantic services increase the accuracy of the references provided by the users to answer given questions?
Question 3: Do the social and semantic services increase overall satisfaction of using the digital library?
Question 4: Which services, i.e., semantic, social, or recommendations, are found to be most useful by the end users?
Question 5: Do social and semantic services improve the information retention?
Digital Enterprise Research Institute www.deri.ie
Evaluation Dimensions
Impact of social and semantic features: One group of users should use a library with and the other group
without semantic and social services.
Half of the evaluation participants use a semantic digital library, the other (control) group used a popular, classic digital library.
Learning curve: Does the particular digital library facilitate users in learning it, and
hence improving the quality of their work, over the time?
To answer this question we engaged participants in a number of tasks spread over longer period of time.
Types of enhancing services: three types: search and browsing on semantics, collaborative
services, recommendation services (reasoning on semantics).
Perform three QA tasks; during each task a new type of enhanced information discovery features was introduced.
22
Digital Enterprise Research Institute www.deri.ie
Evaluation Setup
Two vanilla versions of digital libraries: DSpace - open source “classic” digital library
JeromeDL - open source semantic digital library
Database: noise: 529 articles from http://library.deri.ie/ and http://
books.deri.ie/
references database: 35 articles on Internet psychology
Evaluation site: opened Dec 18th, 2007 and Feb 7th, 2008
advertised on national (Poland) and international social networks
Participants: 59 initiated, 26 completed
23
Digital Enterprise Research Institute www.deri.ie
Evaluation Scenario
Pre-evaluation questionnaire - demographics Initial task: getting to know your library Core tasks: question-answering:
3 rounds, max 45 minutes each
pool of 7 questions from the Internet psychology
up to 300 words answer
unlimited number of supporting references
at least 6 hour breaks
Memory task: question-answering after a month Questionnaire after each task: measuring user satisfaction
24
Digital Enterprise Research Institute www.deri.ie
Evaluation Scenario
25
Digital Enterprise Research Institute www.deri.ie
Satisfaction metrics
Solution-related satisfaction metrics easy to use vs hard to use
complex, mind boggling vs simple,clearly organized
hard to master, unintuitive vs intuitive, straight forward
boring vs interesting
ugly, unattractive vs attractive
useless vs useful, handy.
Task-related satisfaction metrics hard to understand vs easy to understand
hard to execute vs easy to execute
hard to master, unintuitive vs intuitive, straight forward.
Overall satisfaction metrics
26
Digital Enterprise Research Institute www.deri.ie
Q1: Quality of answers
27
Digital Enterprise Research Institute www.deri.ie
Q2: Accuracy of provided references
28
Prec
isio
n
R
ecal
l
Digital Enterprise Research Institute www.deri.ie
Q2: Accuracy of provided references
29
Recall
Precision
Digital Enterprise Research Institute www.deri.ie
Q3: Overall satisfaction (all tasks)
30
Digital Enterprise Research Institute www.deri.ie
Q3: Overall satisfaction (search tasks)
31
Digital Enterprise Research Institute www.deri.ie
Q4: Most useful type of services
32
NLQTTM
ExhibitMBB
col. browsingbookmarks/SSCF
blogranking
bookmarks rec.resource rec. 19.32
15.8312.47
13.4417.76
10.282.68
9.723.62
6.42
Digital Enterprise Research Institute www.deri.ie
Q5: Knowledge Retention
33
Quality of answers: JeromeDL - 2.78, DSpace - 2.44 Correct References: JeromeDL - 6, DSpace - 1 Satisfaction:
Digital Enterprise Research Institute www.deri.ie
Would like to continue using this library ?
JeromeDL DSpace
Digital Enterprise Research Institute www.deri.ie
Would like to continue using this library ?
84.62%
46.15%
JeromeDL DSpace
Digital Enterprise Research Institute www.deri.ie
Conclusions
Quality and accuracy of answers is slightly higher when using semantic and social features
User satisfaction is much higher when using JeromeDL
Further research should focus more on: collaborative and recommendation services - they were
perceived to be the most welcome hiding semantic features behind the scene, e.g., making
simple search much “smarter” Lowering number and diversity of solutions, or
introducing gradual engagemement
35
Digital Enterprise Research Institute www.deri.ie
Related Information
Complete report on JeromeDL evaluation:
http://library.deri.ie/resource/ARfuVUi8
More information about Semantic Digital Libraries:
http://semdl.corrib.org/Book/
http://semdl.corrib.org/Tutorial/
JeromeDL home page:
http://www.jeromedl.org/
JeromeDL users mailing list:
Sebastian Ryszard KrukDERI NUI Galway
36