+ All Categories
Home > Technology > Ecdl2008 Jeromedl Evaluation Long

Ecdl2008 Jeromedl Evaluation Long

Date post: 21-Jan-2015
Category:
Upload: sebastian-kruk
View: 1,579 times
Download: 2 times
Share this document with a friend
Description:
Presentation I gave at ECDL2008
Popular Tags:
50
Copyright 2007 Digital Enterprise Research Institute. All rights reserved. www.deri.org Digital Enterprise Research Institute www.deri.ie Evaluation of Semantic and Social Technologies for Digital Libraries Sebastian R. Kruk, Ewelina Kruk, Katarzyna Stankiewicz
Transcript
Page 1: Ecdl2008 Jeromedl Evaluation Long

Chapter Copyright 2007 Digital Enterprise Research Institute. All rights reserved. www.deri.org

Digital Enterprise Research Institute www.deri.ie

Evaluation of Semantic and Social Technologies for Digital Libraries

Sebastian R. Kruk, Ewelina Kruk, Katarzyna Stankiewicz

Page 2: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Outline of presentation

What are Semantic Digital Libraries Overview of JeromeDL Semantic and social information discovery solutions Evaluation goals and setup Evaluation results Conclusions

2

Page 3: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Semantic Digital Libraries - Motivation

• How to integrate and search information from different sources? • How to share and interconnect knowledge among people?

3

Page 4: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

JeromeDL - Semantic Digital Library

Open source prototype of Semantic DL research; jointly developed by Gdansk University of Technology, Poland and DERI

Social Semantic Information Spaces Semantic description (interconnected metadata)

Annotations provided by users (social metadata)

Collaborative search and browsing (interface)

Features Search and browsing based on semantics empowers users

Users contribute to the classification process

Users can understand community driven annotations

Users enhance digital content using blogs, wikis on the side

Library can interact with other Internet services

4

Page 5: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

9

Metadata and Services in JeromeDL

5

Page 6: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Search and browse on semantics

Natural language templates allows to perform complex queries using natural language

Dynamic Collections easily extensible

Resource-based Recommendations customizable view of recommendations

TagsTreeMaps clustered tags rendered with treemaps algorithm

zoomable interface paradigm

MultiBeeBrowse collaborative and adaptive approach to perform complex browsing

Exhibit (SIMILE, MIT) powerful faceted filtering

6

Page 7: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Natural language templates

Find articles related to mission in the context of aerospace

...QueryTemplates(Regular Expressions)

English Portuguese

Aerospacemission

skos:relatedskos:narrower

resultsmarcont:hasKeyword marcont:hasDomain

SELECT * FROM ....

Page 8: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Resource-based Recommendations

Page 9: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Resource-based Recommendations

Library resource

Page 10: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Resource-based Recommendations

Library resource

hasKeyword

hasDomain

hasCreator

...

Page 11: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Resource-based Recommendations

Library resource

hasKeyword

hasDomain

hasCreator

A

C

D

E

F

Step 1: Find similar resources

G

...

E

C

B

A

Page 12: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Resource-based Recommendations

Library resource

hasKeyword

hasDomain

hasCreator

AE

Step 1: Find similar resources

Step 2: Rank and filter according to user’s settings

...

EBA

Page 13: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Resource-based Recommendations

Library resource

hasKeyword

hasDomain

hasCreator

AE

Step 1: Find similar resources

Step 2: Rank and filter according to user’s settings

...

EBA

summary (max. 3)

Page 14: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Browsing on semantics

TagsTreeMaps filtering based on clustered tags

using treemaps to present the tag space

zoomable interface paradigm

MultiBeeBrowse collaborative browsing

allows to perform complex browsing operations

user can overview browsing context and look up browsing history

Exhibit (SIMILE, MIT) powerful faceted filtering

9

Page 15: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

TTM - Filtering with hierarchical tags

Problems with Tag Clouds: information overload (for large tag clouds)

cannot carry structure and/or semantics

querying model: only conjunctive queries

Solution: limits the information overload

– clustering tagging space

– limiting popularity range

zoomable browser on the tagging space

selecting multiple tags– fulltext filtering - easy highlight matching tags

– optional conjunctive (AND) and union (OR) mode

defined interfaces for delivering processors in the pipeline (e.g., clustering, filtering, coloring)

10

Page 16: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

TTM - Filtering with hierarchical tags

11

Page 17: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Adaptive

Presenting results human-readable names of concepts

type-specific rendering

limiting information overload with stretch-text

Refining queries in-situ each concept is seed to new query

different actions based on concept type

Suggesting properties and concepts most frequently used

recently used

Accessible predicated names human-readable names of properties

support for inverted properties

12

Page 18: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Collaborative

Nowadays people share: photos, music, links, etc. - why not queries ?

Collaborative filtering solution adapted for sharing browsing experience based on Social Semantic Collaborative Filtering service

users can tag/annotate their queries

users can share queries with their friends

13

Page 19: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Zoomable

Helping users with different problems Finding results

Going back and forth in the refinement process

Overview of current browsing context

Replaying previous queries

4 views: Basic browsing view

Structured history view

HoneyComb view

Life-long history view

14

Page 20: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Structured

15

Page 21: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Structured

15

Page 22: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Structured

15

Page 23: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Structured

15

Page 24: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - Structured

15

Page 25: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - HoneyComb View

16

Page 26: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - HoneyComb View

16

Page 27: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - HoneyComb View

16

Page 28: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - HoneyComb View

16

Page 29: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

MultiBeeBrowse - HoneyComb View

16

Page 30: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Social Services in JeromeDL

• Involve users into sharing knowledge

– Blogs – comments and discussions about documents and

resources

– Tagging – collaborative classification

– Wikis – collaboratively edited additional descriptions, such as

summaries and interesting facts

• Preserve knowledge for future use

– users can learn from experience of others instantly

– recommend new, interesting resources based on users’ profiles

17

Page 31: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

What is Social Semantic Collaborative Filtering?

• Goal: to enhance individual bookmarks with shared knowledge within a community

• Users annotate catalogues of bookmarks with semantic information taken from DMoz or WordNet vocabularies

• Catalogs can include (transclusion) friend's catalogues

• Access to catalogues can be restricted with social networking-based polices

• SSCF delivers:– Community-oriented, semantically-rich taxonomies

– Information about a user's interest

– Flows of expertise from the domain expert

– Recommendations based on users previous actions

– Support for SIOC metadata

18

Page 32: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

knows

include

bookmark

Social Semantic Collaborative Filtering

19

recommend

Page 33: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Scope of Evaluation

20

Page 34: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Questions for Evaluation

Question 1: Do the social and semantic services increase the quality of the answers provided by the users in response to given problems?

Question 2: Do the social and semantic services increase the accuracy of the references provided by the users to answer given questions?

Question 3: Do the social and semantic services increase overall satisfaction of using the digital library?

Question 4: Which services, i.e., semantic, social, or recommendations, are found to be most useful by the end users?

Question 5: Do social and semantic services improve the information retention?

Page 35: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Evaluation Dimensions

Impact of social and semantic features: One group of users should use a library with and the other group

without semantic and social services.

Half of the evaluation participants use a semantic digital library, the other (control) group used a popular, classic digital library.

Learning curve: Does the particular digital library facilitate users in learning it, and

hence improving the quality of their work, over the time?

To answer this question we engaged participants in a number of tasks spread over longer period of time.

Types of enhancing services: three types: search and browsing on semantics, collaborative

services, recommendation services (reasoning on semantics).

Perform three QA tasks; during each task a new type of enhanced information discovery features was introduced.

22

Page 36: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Evaluation Setup

Two vanilla versions of digital libraries: DSpace - open source “classic” digital library

JeromeDL - open source semantic digital library

Database: noise: 529 articles from http://library.deri.ie/ and http://

books.deri.ie/

references database: 35 articles on Internet psychology

Evaluation site: opened Dec 18th, 2007 and Feb 7th, 2008

advertised on national (Poland) and international social networks

Participants: 59 initiated, 26 completed

23

Page 37: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Evaluation Scenario

Pre-evaluation questionnaire - demographics Initial task: getting to know your library Core tasks: question-answering:

3 rounds, max 45 minutes each

pool of 7 questions from the Internet psychology

up to 300 words answer

unlimited number of supporting references

at least 6 hour breaks

Memory task: question-answering after a month Questionnaire after each task: measuring user satisfaction

24

Page 38: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Evaluation Scenario

25

Page 39: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Satisfaction metrics

Solution-related satisfaction metrics easy to use vs hard to use

complex, mind boggling vs simple,clearly organized

hard to master, unintuitive vs intuitive, straight forward

boring vs interesting

ugly, unattractive vs attractive

useless vs useful, handy.

Task-related satisfaction metrics hard to understand vs easy to understand

hard to execute vs easy to execute

hard to master, unintuitive vs intuitive, straight forward.

Overall satisfaction metrics

26

Page 40: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q1: Quality of answers

27

Page 41: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q2: Accuracy of provided references

28

Prec

isio

n

R

ecal

l

Page 42: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q2: Accuracy of provided references

29

Recall

Precision

Page 43: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q3: Overall satisfaction (all tasks)

30

Page 44: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q3: Overall satisfaction (search tasks)

31

Page 45: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q4: Most useful type of services

32

NLQTTM

ExhibitMBB

col. browsingbookmarks/SSCF

blogranking

bookmarks rec.resource rec. 19.32

15.8312.47

13.4417.76

10.282.68

9.723.62

6.42

Page 46: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Q5: Knowledge Retention

33

Quality of answers: JeromeDL - 2.78, DSpace - 2.44 Correct References: JeromeDL - 6, DSpace - 1 Satisfaction:

Page 47: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Would like to continue using this library ?

JeromeDL DSpace

Page 48: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Would like to continue using this library ?

84.62%

46.15%

JeromeDL DSpace

Page 49: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Conclusions

Quality and accuracy of answers is slightly higher when using semantic and social features

User satisfaction is much higher when using JeromeDL

Further research should focus more on: collaborative and recommendation services - they were

perceived to be the most welcome hiding semantic features behind the scene, e.g., making

simple search much “smarter” Lowering number and diversity of solutions, or

introducing gradual engagemement

35

Page 50: Ecdl2008 Jeromedl Evaluation Long

Digital Enterprise Research Institute www.deri.ie

Related Information

Complete report on JeromeDL evaluation:

http://library.deri.ie/resource/ARfuVUi8

More information about Semantic Digital Libraries:

http://semdl.corrib.org/Book/

http://semdl.corrib.org/Tutorial/

JeromeDL home page:

http://www.jeromedl.org/

JeromeDL users mailing list:

[email protected]

Sebastian Ryszard KrukDERI NUI Galway

[email protected]

36


Recommended